Business Associate

Senior Analyst - Azure Data Engineer - Capability BOI Risk Management Team.

 

End-to-End Data Ingestion & Integration:

Design, develop, and implement scalable data ingestion pipelines to extract, load, and process structured and unstructured data from multiple sources, including on-prem databases, APIs, cloud storage into Databricks.

Data Transformation & Optimization:

§Perform complex data transformations, including joins, aggregations, deduplication, and filtering using PySpark, SQL, and Python, ensuring high-performance processing and efficiency. §Implement Medallion Architecture (Bronze, Silver, Gold) for structured data movement. §Establish data validation, cleansing, and standardization techniques to maintain data integrity and accuracy across the pipeline. §Monitor data pipelines for failures, optimize query performance, and implement error-handling mechanisms to ensure reliability.

Version Control & CI/CD Automation:

Utilize GitHub for source control, versioning, and collaborative development. Implement CI/CD pipelines to automate deployments, ensuring seamless integration of code changes in Databricks Workflows.

Collaboration & Stakeholder Communication:

Work closely with data engineers, analysts, and business teams to gather requirements, design solutions, and deliver insights-driven data pipelines. Document data workflows, best practices, and technical specifications for efficient handover and knowledge sharing.

Qualifications: Graduate or a post graduate degree in Information Technology or a related field from Tier 1/Tier 2 institute 3-6 years of experience as an Azure Data Engineer, with a focus on Databricks, data pipelines, and cloud-based data management. Strong expertise in Azure Databricks, including ETL development, data transformations, and workflow orchestration. Extensive experience in designing and implementing scalable data models optimized for analytics and reporting. Proficiency in Databricks Workflows, Unity Catalog, and Spark to manage and govern data efficiently.   •Hands-on experience with Azure Data Factory, Synapse Analytics, and Azure Data Lake for seamless data movement and transformation. •Strong knowledge of API integrations, connecting Databricks with external systems such as data warehouses, REST APIs, and cloud storage solutions. •Experience in building and managing robust data pipelines, ensuring high-quality structured datasets for business intelligence. •Proficiency in SQL, PySpark, and Python for advanced data transformations and processing. •Experience in implementing Medallion Architecture to structure data movement across Bronze, Silver, and Gold layers. •Familiarity with CI/CD for Databricks, using GitHub or Azure DevOps for version control and automated deployments. •Willing to work hard, enthusiastic, and self-motivated •Strong commitment, multi-tasking, and strong work ethics