Databricks Engineer

North Macedonia; Romania

Encora

Encora provides its clients with tailored innovation software engineering solutions across a wide range of leading-edge technologies.

View all jobs at Encora

Apply now Apply later

Ā 

Ā 

Important InformationĀ 

  • Experience: More than 5 yearsĀ 
  • Job Mode: Full-timeĀ 
  • Work Mode: Hybrid

Ā 

Job Summary

Ā 

We are looking for a Data Engineer with deep expertise in building, optimizing and managing scalable data pipelines using Azure Databricks, Spark (SQL, PySpark) and related Azure services. The ideal candidate will play a critical role in designing robust data workflows to support ingestion, transformation, and publication of data for analytics and business integrations, with a strong emphasis on performance, scalability, and governance.

Ā 

Responsibilities and Duties

Data Pipeline Development

  • Design and implement end-to-end data pipelines using Azure Databricks.

  • Write efficient Spark SQL and PySpark code to transform data with integrity and accuracy.

  • Automate workflows using Databricks Workflows, Azure Data Factory, or Apache Airflow.

Data Ingestion & Transformation

  • Build scalable ingestion processes for structured, semi-structured, and unstructured data.

  • Develop robust transformation logic and leverage Delta Lake for versioning and incremental loads.

Data Publishing & Integration

  • Publish clean and transformed data to Azure Data Lake, making it available for analytics.

  • Define and document best practices for data pipeline development and maintenance.

Data Governance & Security

  • Apply governance policies using Unity Catalog, ensuring access control and metadata management.

  • Implement encryption and RBAC standards across data platforms for secure operations.

Performance Tuning & Optimization

  • Optimize Spark workloads for speed and cost-efficiency through partitioning, caching, and tuning.

  • Monitor data pipelines continuously, resolving bottlenecks and scaling as needed.

Automation & Monitoring

  • Automate infrastructure and deployments using Terraform or similar tools.

  • Set up proactive monitoring using Azure Monitor and Databricks native capabilities.

Qualifications and Skills

  • Bachelor’s or Master’s degree in Computer Science, Data Engineering, or related field.

  • 5+ years of experience in data engineering with a focus on Azure and Databricks ecosystems.

  • Strong communication and documentation skills.

  • Ability to work independently and collaboratively within cross-functional teams.

Role-Specific Requirements

  • Proven experience designing ETL/ELT pipelines in Azure Databricks.

  • Expertise in Spark SQL and PySpark development.

  • Hands-on with Databricks Workflows, Delta Lake, and Azure Data Lake.

  • Deep understanding of data governance, especially using Unity Catalog.

  • Proficient in performance tuning Spark jobs and optimizing cloud costs.

Technologies

  • Azure Databricks, Apache Spark, PySpark, SQL

  • Azure Data Lake, Azure Blob Storage, Azure Synapse

  • Delta Lake, Unity Catalog

  • Azure Data Factory, Apache Airflow

  • Terraform, Azure Monitor

Skillset Competencies

  • Data pipeline architecture and design

  • Cloud-based data engineering (Azure)

  • Performance and cost optimization

  • Automation and DevOps in data environments

  • Data governance and security compliance

  • Effective troubleshooting and monitoring of data workflows

About EncoraĀ 

Encora is a trusted partner for digital engineering and modernization, working with some of the world’s leading enterprises and digital-native companies. With over 9,000 experts in 47+ offices worldwide, Encora offers expertise in areas such as Product Engineering, Cloud Services, Data & Analytics, AI & LLM Engineering, and more. At Encora, hiring is based on skills and qualifications, embracing diversity and inclusion regardless of age, gender, nationality, or background.Ā 

Ā 

Apply now Apply later

* Salary range is an estimate based on our AI, ML, Data Science Salary Index šŸ’°

Job stats:  0  0  0
Category: Engineering Jobs

Tags: Airflow Architecture Azure Computer Science Databricks Data governance Data pipelines DevOps ELT Engineering ETL LLMs Pipelines PySpark Security Spark SQL Terraform Unstructured data

Region: Europe

More jobs like this