Lead Data Engineer

Kolkata, West Bengal, India

Full Time Senior-level / Expert USD 28K - 67K *

Adani Group

A leading integrated business conglomerate enriching lives, creating sustainable value and empowering India through #GrowthWithGoodness.

View all jobs at Adani Group

Apply now Apply later

Posted 6 hours ago

Implement enterprise data lake, develop data transformation & conversion strategies from multiple source systems, federated Datahouses/Data Lakes with data retention & archiving strategies along with regulatory requirements
Build data pipelines to implement Analytics. AI/ML use cases in real time and batch
Create data consumption layer for BI/visualization tools
Establish data governance, data security and data cleansing procedures to ensure Data Quality & Compliance
Mange cloud consumption cost through optimization of services

Skill and Experience

4+ years of experience in cloud platforms of Azure\GCP
3+ experience in Databricks
Experience in building data pipelines (batch and real time) and handling variety of data sources including semi-structured and unstructured data sources (image/video, GPS. IoT)
Strong expertise in ETL/ELT processes, data lake, Datawarehouse, data mesh
Excellent knowledge of python, SQL, cloud DWHs (Synapse, BigQuery), managed DBs (CloudSQL, Azure DBs), PostgreSQL
Hands on experience in Databricks (DLT, CDC, Lakeflow Connect), Unity catalogue
Knowledge of open table formats (Delta, Iceberg) and tools (Trino, Flink, Presto)
Good knowledge of scheduling and orchestration tools like Airflow, Compose
Expertise in Azure services like ADLS, ADF, Functions, Event Grid, Stream Analytics
Expertise in GCP services like GCS, Dataflow, Dataproc, Cloud Functions, Cloud Run
Experience in developing and deploying API endpoints on API gateways/APIGEE
Experience in handling streaming data and messaging tools (Kafka, Pub/Sub, Service Bus Messaging, Event Hub)
Experience in deploying workloads on managed Kubernetes clusters
Strong analytical & problem-solving skills with the ability to think critically & creatively about complex business problems
Ability to work independently & collaboratively and to manage multiple projects & priorities simultaneously
Strong communication skills
High level of collaboration & stakeholder management

Education Qualification

Bachelor’s degree in engineering
Professional certification in Azure, GCP, Databricks

Apply now Apply later

* Salary range is an estimate based on our AI, ML, Data Science Salary Index 💰

Job stats: 0 0 0

Categories: Engineering Jobs Leadership Jobs

Tags: Airflow APIs Azure BigQuery Databricks Dataflow Data governance Data pipelines Dataproc Data quality ELT Engineering ETL Flink GCP Kafka Kubernetes Machine Learning Pipelines PostgreSQL Python Security SQL Streaming Unstructured data

Region: Asia/Pacific

Country: India

More jobs like this

« Back to job search To the top ↑

Explore more career opportunities

Find even more open roles below ordered by popularity of job title or skills/products/technologies used.