Databricks Engineer
North Macedonia; Romania
Encora
Encora provides its clients with tailored innovation software engineering solutions across a wide range of leading-edge technologies.Ā
Ā
Important InformationĀ
- Experience: More than 5 yearsĀ
- Job Mode: Full-timeĀ
- Work Mode: Hybrid
Ā
Job Summary
Ā
We are looking for a Data Engineer with deep expertise in building, optimizing and managing scalable data pipelines using Azure Databricks, Spark (SQL, PySpark) and related Azure services. The ideal candidate will play a critical role in designing robust data workflows to support ingestion, transformation, and publication of data for analytics and business integrations, with a strong emphasis on performance, scalability, and governance.
Ā
Responsibilities and Duties
Data Pipeline Development
-
Design and implement end-to-end data pipelines using Azure Databricks.
-
Write efficient Spark SQL and PySpark code to transform data with integrity and accuracy.
-
Automate workflows using Databricks Workflows, Azure Data Factory, or Apache Airflow.
Data Ingestion & Transformation
-
Build scalable ingestion processes for structured, semi-structured, and unstructured data.
-
Develop robust transformation logic and leverage Delta Lake for versioning and incremental loads.
Data Publishing & Integration
-
Publish clean and transformed data to Azure Data Lake, making it available for analytics.
-
Define and document best practices for data pipeline development and maintenance.
Data Governance & Security
-
Apply governance policies using Unity Catalog, ensuring access control and metadata management.
-
Implement encryption and RBAC standards across data platforms for secure operations.
Performance Tuning & Optimization
-
Optimize Spark workloads for speed and cost-efficiency through partitioning, caching, and tuning.
-
Monitor data pipelines continuously, resolving bottlenecks and scaling as needed.
Automation & Monitoring
-
Automate infrastructure and deployments using Terraform or similar tools.
-
Set up proactive monitoring using Azure Monitor and Databricks native capabilities.
Qualifications and Skills
-
Bachelorās or Masterās degree in Computer Science, Data Engineering, or related field.
-
5+ years of experience in data engineering with a focus on Azure and Databricks ecosystems.
-
Strong communication and documentation skills.
-
Ability to work independently and collaboratively within cross-functional teams.
Role-Specific Requirements
-
Proven experience designing ETL/ELT pipelines in Azure Databricks.
-
Expertise in Spark SQL and PySpark development.
-
Hands-on with Databricks Workflows, Delta Lake, and Azure Data Lake.
-
Deep understanding of data governance, especially using Unity Catalog.
-
Proficient in performance tuning Spark jobs and optimizing cloud costs.
Technologies
-
Azure Databricks, Apache Spark, PySpark, SQL
-
Azure Data Lake, Azure Blob Storage, Azure Synapse
-
Delta Lake, Unity Catalog
-
Azure Data Factory, Apache Airflow
-
Terraform, Azure Monitor
Skillset Competencies
-
Data pipeline architecture and design
-
Cloud-based data engineering (Azure)
-
Performance and cost optimization
-
Automation and DevOps in data environments
-
Data governance and security compliance
-
Effective troubleshooting and monitoring of data workflows
About EncoraĀ
Encora is a trusted partner for digital engineering and modernization, working with some of the worldās leading enterprises and digital-native companies. With over 9,000 experts in 47+ offices worldwide, Encora offers expertise in areas such as Product Engineering, Cloud Services, Data & Analytics, AI & LLM Engineering, and more. At Encora, hiring is based on skills and qualifications, embracing diversity and inclusion regardless of age, gender, nationality, or background.Ā
Ā
* Salary range is an estimate based on our AI, ML, Data Science Salary Index š°
Tags: Airflow Architecture Azure Computer Science Databricks Data governance Data pipelines DevOps ELT Engineering ETL LLMs Pipelines PySpark Security Spark SQL Terraform Unstructured data
More jobs like this
Explore more career opportunities
Find even more open roles below ordered by popularity of job title or skills/products/technologies used.