Azure Databricks

Data Engineer proficient in Azure Databricks to build and maintain robust data pipelines

seeking a highly motivated and skilled Azure Databricks Engineer to join our growing data engineering team. The ideal candidate will have a strong background in data engineering principles and hands-on experience with Azure Databricks, Spark, and related technologies. You will play a key role in designing, building, and maintaining our data pipelines and analytics platform on Azure.

Responsibilities:

• Design, develop, and maintain scalable and reliable data pipelines using Azure Databricks, Spark, and other relevant Azure services (e.g., Azure Data Factory, Azure Blob Storage, Azure Data Lake Storage Gen2).
• Develop and optimize Spark applications using PySpark, Scala, or SQL.
• Implement data ingestion, transformation, and processing logic using Databricks notebooks, jobs, and workflows.
• Work with various data formats (e.g., Parquet, Avro, JSON, CSV) and data storage solutions.
• Implement data quality checks and monitoring to ensure data accuracy and reliability.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver effective solutions.   
• Optimize performance of Databricks workloads and troubleshoot performance bottlenecks.   
• Implement and maintain CI/CD pipelines for Databricks deployments.
• Contribute to the development of best practices and standards for Azure Databricks development.
• Stay up-to-date with the latest Azure Databricks features and advancements.
• Potentially manage and administer the Azure Databricks environment.

Qualifications:

• Bachelor's degree in Computer Science, Engineering, or a related field (or equivalent experience).
• Proven experience in data engineering and working with large datasets.   
• Strong hands-on experience with Azure Databricks and Apache Spark.
• Proficiency in at least one programming language: Python (PySpark preferred), Scala, or SQL.
• Experience with data warehousing concepts and data modeling.
• Familiarity with Azure cloud services, including Azure Data Factory, Azure Blob Storage, Azure Data Lake Storage Gen2, and other relevant services.
• Experience with version control systems (e.g., Git).
• Understanding of CI/CD principles and experience with related tools.
• Excellent problem-solving and analytical skills.
• Strong communication and collaboration skills.

Preferred Qualifications:

• Azure certifications (e.g., Azure Data Engineer Associate, Azure Solutions Architect Expert).   
• Experience with Delta Lake.
• Experience with Databricks Delta Live Tables (DLT).
• Experience with stream processing technologies (e.g., Kafka, Spark Streaming).
• Experience with data governance and security best practices.
• Experience with infrastructure-as-code tools (e.g., Terraform, ARM templates).