Senior/Lead Data Engineer.

Location: Gurugram,Haryana,India

About us 

We turn customer challenges into growth opportunities.  

Material is a global strategy partner to the world’s most recognizable brands and innovative companies. Our people around the globe thrive by helping organizations design and deliver rewarding customer experiences.  

We use deep human insights, design innovation and data to create experiences powered by modern technology. Our approaches speed engagement and growth for the companies we work with and transform relationships between businesses and the people they serve. 

Srijan, a Material company, is a renowned global digital engineering firm with a reputation for solving complex technology problems using their deep technology expertise and leveraging strategic partnerships with top-tier technology partners. Be a part of an Awesome Tribe

Role- Senior/Lead Data Engineer (Azure, Databricks, Data Factory, Data Fabric) 

Job Responsibilities:

• Design and Develop Data Pipelines: Development and optimisation of scalable data pipelines on Azure Databricks, utilising both batch and real-time processing techniques. Ensure seamless integration with Azure Data Factory for orchestration and workflow management.
• Cloud Data Architecture: Collaborate closely with the Data Architecture team on development and implement robust data architectures that align with business needs and optimise performance, scalability, and cost-efficiency in the Azure environment.
• Data Pipeline Optimisation: Continuously monitor, enhance, and optimise data pipelines for performance, cost, and reliability, ensuring the efficient processing and storage of large datasets.
• Collaboration with Cross-functional Teams: Work with data engineering teams, analysts, and business stakeholders to understand their data requirements and ensure that the data platform meets those needs. Enable self-service analytics through well-designed data models and solutions.
• Documentation and Knowledge Sharing: Contribute to internal documentation and knowledge-sharing practices. Provide technical mentorship to junior engineers to improve team skills and performance.
• Azure Data Fabric Familiarity: Leverage your knowledge of Azure Data Fabric (or willingness to quickly learn) to manage end-to-end data movement, orchestration, and governance across various cloud platforms and on-prem systems. Ensure seamless data integration across hybrid environments.
• Databricks Platform Expertise: Use your deep expertise in Azure Databricks to build and manage data models, create data processing pipelines, and integrate with other Azure services such as Azure Data Lake and Synapse for data storage and analytics.

Required Skills and Qualifications:

• Experience in Azure Databricks: 5+ years of hands-on experience with Azure Databricks, including Spark, Delta Lake, and other relevant Databricks tools. Expertise in optimising Databricks clusters and managing big data workloads efficiently.
• Proficiency in Azure Data Factory: Strong experience in designing and orchestrating data pipelines using Azure Data Factory, including integrating with other Azure services for seamless data flow.
• Familiarity with Azure Data Fabric: Working knowledge or eagerness to learn Azure Data Fabric to manage cross-platform data orchestration, governance, and security.
• Advanced Data Engineering Skills: Extensive experience in data engineering, including data pipeline development, ETL processes, and working with large datasets. Proven ability to implement best practices in data quality, monitoring, and testing.
• Cloud Architecture Design: Experience designing and implementing cloud-based data architectures that leverage Azure’s ecosystem, including Data Lake, Synapse, and Azure Storage.
• Strong SQL and Data Modelling: Expertise in SQL and data modelling techniques to create optimised data structures, tables, and views. Understanding of both transactional and analytical data modelling.
• Collaboration and Communication: Strong ability to collaborate with cross-functional teams and effectively communicate complex technical concepts to both technical and non-technical stakeholders.
• Cost Optimisation: Proven experience optimising data engineering processes and Azure resources for cost reduction and performance improvements, particularly in large-scale cloud environments.

Preferred Skills:

• Familiarity with Data Lakehouse architectures and working with tools like Delta Lake, One Lake, etc.
• Familiarity with the Azure ecosystem for data engineering, specifically for end-to-end data integration and ETL pipelines.
• Knowledge of PySpark, strong SQL skills, and basic Python knowledge is mandatory.
  Familiarity with Databricks and its integration with the broader Azure data platform.

Who We Are & What We Care About 

• Material is a global company and we work with best-of-class brands worldwide. We also create and launch new brands and products, putting innovation and value creation at the center of our practice. Our clients are in the top of their class, across industry sectors from technology to retail, transportation, finance and healthcare. 
• Material employees join a peer group of exceptionally talented colleagues across the company, the country, and even the world. We develop capabilities, craft and leading-edge market offerings across seven global practices including strategy and insights, design, data & analytics, technology and tracking. Our engagement management team makes it all hum for clients.  
• We prize inclusion and interconnectedness. We amplify our impact through the people, perspectives, and expertise we engage in our work. Our commitment to deep human understanding combined with a science & systems approach uniquely equips us to bring a rich frame of reference to our work. 
• A community focused on learning and making an impact. Material is an outcomes focused company. We create experiences that matter, create new value and make a difference in people's lives.