Lead – Data Engineering

Generic Accountability

§  Developing and deploying data engineering workloads in Databricks on the Azure cloud infrastructure, managing and monitoring performance of these workloads, and ensuring security, availability, and scalability of cloud resources.

 

5. JOB CONTEXT

(Specific accountabilities unique for the role which are not covered in Section 4)

Specific Accountability

·        Work with the business lines to understand the data ingestion and cloud data engineering requirements

·        Work with the data modellers to translate the same into cloud/on-prem data engineering workloads design and then develop the same into appropriate engineering language – Databricks pipelines and ADF pipelines.

·        Very strong hands-on skills and at least 3 years experience in developing databricks and Azure Data Factory pipelines

·        Follow the principle of medallion architecture (3 layers of data in Azure data lake) of getting the data in right data zones basis the requirement

·        Work and collaborate in multi-disciplinary Agile teams, adopting Agile spirit, methodology and tools

·        Help code and scale applications in a multi-cloud environment integrated with on-premise

·        Work with the platform lead to spot out and remediate the potential operational risks of the platform

Requirements

Minimum Qualification

·         

·        Overall 5-7 years of experience in data engineering and transformation on Cloud

·        3+ Years of Very Strong Experience in Azure Data Engineering, Databricks

·        Expertise in supporting/developing Data warehouse workloads at enterprise level

·        Experience in pyspark is required – developing and deploying the workloads to run on the Spark distributed computing

·        Candidate must possess at least a Graduate or bachelor’s degree in Computer Science/Information Technology, Engineering (Computer/Telecommunication) or equivalent.

·        Cloud deployment: Preferably Microsoft azure

·        Experience in implementing the platform and application monitoring using Cloud native tools

Experience in implement application self-healing solutions through proactive and reactive automated measures