Sr Data Engineer x 4

Job Title: Senior Data Engineer (Azure Databricks & ADLS)
Location: Remote
Contract Duration: May – End of the Year

Job Overview:
We are seeking a highly skilled Senior Data Engineer to design, build, and maintain scalable and efficient data pipelines on Azure Databricks and Azure Data Lake Storage (ADLS). The ideal candidate will have expertise in Azure Data Factory (ADF), PySpark, Databricks Auto Loader, Delta Lake, and Change Data Capture (CDC) to optimize both real-time and batch data processing. This role requires strong Azure SQL skills, data governance expertise, and the ability to mentor junior engineers while collaborating with cross-functional teams.

Key Responsibilities:

• Data Pipeline Development: Design, build, and optimize scalable data pipelines using Azure Databricks, ADLS, and Azure Data Factory (ADF).
  
• Data Ingestion & Transformation: Implement robust data ingestion and transformation solutions using PySpark, Databricks Auto Loader, and Delta Lake to ensure high performance and reliability.
  
• Change Data Capture (CDC): Develop and optimize real-time CDC mechanisms to efficiently track and process data changes across multiple systems.
  
• Azure SQL & Query Optimization: Utilize Azure SQL for structured data management, optimizing queries, indexing, and partitioning for efficiency.
  
• Data Governance & Security: Implement role-based access controls, encryption, and compliance policies to secure data and meet regulatory standards.
  
• Monitoring & Performance Tuning: Continuously monitor, troubleshoot, and fine-tune ADF pipelines to optimize workflow performance and minimize latency.
  
• Collaboration & Stakeholder Engagement: Work closely with data engineers, analysts, and business stakeholders to understand data requirements and deliver scalable, high-quality solutions.
  
• CI/CD & Automation: Implement CI/CD best practices for data pipelines, ensuring automated deployments and efficient continuous integration/delivery.
  
• Mentorship & Knowledge Sharing: Provide guidance and mentorship to apprentice and junior engineers, fostering a culture of learning and growth.
  
• Continuous Improvement: Stay up to date with emerging Azure technologies, industry trends, and best practices to enhance data engineering solutions and drive innovation.
  
  
  
• Are you always looking to challenge the status quo and optimize data solutions?
  
• Do you love mentoring junior engineers and helping them grow?
  
• Are you excited by the opportunity to work with a consulting company that takes on new and innovative projects?
  

If this sounds like you, keep reading – this opportunity might be your perfect fit!

Requirements

Required Skills & Qualifications:

• 5+ years of experience in data engineering, with a strong background in Azure cloud technologies.
  
• Expertise in Azure Databricks, ADLS, ADF, PySpark, Databricks Auto Loader, and Delta Lake.
  
• Strong experience in real-time and batch data processing, including CDC implementation.
  
• Proficiency in Azure SQL, including query optimization, indexing, and partitioning.
  
• Solid understanding of data governance, security best practices, and compliance requirements.
  
• Experience with CI/CD pipelines and automation for data workflows.
  
• Ability to troubleshoot and optimize complex data pipelines for performance and efficiency.
  
• Strong problem-solving skills, attention to detail, and ability to work independently.
  
• Excellent communication and collaboration skills, with experience mentoring junior engineers.
  

Nice to Have:

• Experience with streaming technologies such as Azure Event Hubs or Kafka.
  
• Familiarity with machine learning data pipelines and MLOps.
  
• Knowledge of Terraform or Infrastructure as Code (IaC) for Azure resources.