Databricks Developer

Position Overview:

We are looking for a skilled Data Engineer with 3 to 6 years of experience working on Azure Databricks and its native services. The ideal candidate will have a strong background in data engineering, ETL processes, and cloud technologies, with hands-on experience designing, building, and optimizing data pipelines. The Data Engineer will collaborate closely with cross-functional teams to support data-driven initiatives and drive continuous improvement in our data architecture and processes.

Key Responsibilities:

• Design, develop, and maintain scalable and efficient data pipelines using Azure Databricks and its native services, such as Azure Data Factory, Azure Synapse Analytics, Azure SQL Database, and Azure Blob Storage.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and implement solutions that meet business needs.
• Perform data profiling, cleansing, and transformation to ensure data quality, consistency, and accuracy throughout the data lifecycle.
• Optimize data pipelines for performance, reliability, and cost-effectiveness, leveraging best practices and automation techniques.
• Implement data security and compliance measures to protect sensitive information and ensure regulatory compliance.
• Monitor and troubleshoot data pipeline issues, proactively identifying and resolving issues to minimize downtime and ensure data availability.
• Document technical designs, data lineage, and system configurations to support knowledge sharing and facilitate collaboration.
• Stay current with industry trends, emerging technologies, and best practices in data engineering and cloud computing, and recommend opportunities for innovation and optimization.

Qualifications:

• Bachelor’s or master’s degree in computer science, Engineering, or a related field.
• 3 to 6 years of experience working as a Data Engineer, with a focus on cloud-based data platforms and services, particularly Azure Databricks and its native services.
• Strong proficiency in SQL, Python, and Spark, with hands-on experience developing ETL processes and data pipelines.
• Experience working with Azure cloud services such as Azure Data Factory, Azure Synapse Analytics, Azure SQL Database, Azure Blob Storage, and Azure Data Lake Storage.
• Knowledge of data modelling techniques, dimensional modelling, and data warehousing concepts.
• Excellent analytical and problem-solving skills, with the ability to troubleshoot complex data issues and implement effective solutions.
• Strong communication and collaboration skills, with the ability to work effectively in a cross-functional team environment.