Lead Data Engineer - PySpark/SQL/AI
111443-IND-HYDERABAD-INTL HYD WF CENTRE BLK B8 Twr-4, India
INR 111K-200K (estimate) Senior-level Full Time
Tasks
- Build batch streaming and real time pipelines
- Collaborate with internal partners on data engineering challenges
- Design Lakehouse architectures
- Design, develop, and run data tooling
- Develop migration assessment planning automation reconciliation testing and cutover
- Discover data problems and report to stakeholders
- Ensure data integrity during migration
- Implement data governance and metadata management
- Implement data mesh and self service analytics
- Lead data engineering initiatives
- Migrate on-premise data to cloud
- Review and analyze data engineering enhancements
- Support enterprise data modernization
Perks/Benefits
- N/A
Skills/Tech-stack
API Integration | Agentic AI | Apache Spark | Azure | Cloud platform | Data Governance | Data Quality | Data Warehousing | Data mesh | DataOps | Databricks | DevSecOps | Docker | ELT | ETL | Generative AI | Google Cloud | Google Cloud Platform | Infrastructure as Code | Kubernetes | Lakehouse | Language Models | Large Language Models | MLOps | Medallion Architecture | Metadata Management | Microsoft Fabric | PySpark | Python | SQL | Self-Service | Self-service analytics | Service analytics | Terraform | “as-code”
Education
Bachelor of Engineering | Bachelor of Science | Master of Science
Roles
Data Engineer | Engineer | Lead | Lead Data | Lead Data Engineer
Related jobs
-
Associate Engineer-AI/ML INR 2165K-3000KAPI Integration | Amazon Web Services | Azure | C++ | CD pipelinesEmployee discounts | Training opportunities | Tuition reimbursement | Wellness benefitsMid-level Full TimeBangalore - Karnataka, India - Kalyani …8h ago
-
Junior BI Developer/Data Engineer - India Home Base INR 300K-300KBig Data | Data Modeling | Data Pipelines | ETL | JavaFlexible remote work | Professional development opportunitiesEntry-level Full TimeBangalore, India8h ago
-
Databricks Data Engineer - Contract-to-Hire INR 1500K-3000KAWS | Apache Spark | Azure | Data Reconciliation | Data ValidationMedical insurance | Paid time off | Professional development trainingSenior-level Full TimePune, MH, India10h ago
-
Senior Databricks Data Engineer INR 1500K-3500KAWS | CI/CD | Databricks | Delta Lake | LakehouseMedical insurance | Paid time off | Performance incentives | Professional development trainingSenior-level Full TimePune, MH, India10h ago
-
Lead AI Specialist (Finance Operations) INR 3264K-5449KAccess Control | Agent Orchestration | Audit Logging | BigQuery | Change ManagementCollaborative culture | Growth opportunities | Work-life balance programsSenior-level Full TimeMumbai, MH, India10h ago
-
Senior-level Full TimeBengaluru, Karnataka, India R10h ago
-
Applied AI ML Associate INR 1400K-2400KAutomation | Data Quality | Data analytics | Data pipeline | DatabaseMid-level Full TimeBengaluru, Karnataka, India11h ago
-
Applied AI ML Associate INR 1400K-2400KCause analysis | Credit Risk | Credit risk modeling | Data Analysis | Data DriftMid-level Full TimeBengaluru, Karnataka, India11h ago
-
AES - DE - Generative AI Solution Architect INR 2500K-4800KAI Agents | Azure | Conversational AI | Copilot Studio | Language ModelsSenior-level Full TimePune, Maharashtra, India11h ago
-
Lead Assistant Manager INR 1500K-2500KApache Airflow | BigQuery | Cloud Composer | Cloud Storage | Cloud platformSenior-level Full TimePune, Maharashtra, India11h ago
-
Cybersecurity Architect III - Offensive Security, Databricks, AI INR 2000K-3500KAWS | Agile methodologies | Artificial Intelligence | Auditability | Continuous DeliverySenior-level Full TimeHyderabad, Telangana, India12h ago
-
Data Engineer Consultant Senior INR 3000K-4000KAgile | Alerting | Auditability | Automated testing | AzureBike to work | Career mentorship | Educational assistance | Employee assistance program | Employee discountSenior-level Full TimeBengaluru, KA, India R13h ago
-
Data Engineer Consultant Principal INR 3000K-4000KAccess Control | Agile | Audit and compliance | Auditability | Automated testingBike to work | Career progression mentorship | Educational assistance | Employee assistance programme | Employee discount schemeSenior-level Full TimeBengaluru, KA, India R13h ago
-
Senior ML Operations Engineer INR 2500K-5000KAI Governance | AWS | Automated retraining | Azure | Azure MLSenior-level Full TimeBengaluru, Karnataka, India14h ago
-
Lead Software Engineer Data Engineer INR 2500K-3500KAWS | Agile | Cloud Native | Code review | Data Performance OptimizationSenior-level Full TimeBengaluru, Karnataka, India14h ago
-
AI Engineer INR 3000K-5000KAI Services | AWS SageMaker | Azure AI | Azure AI Services | Azure DevOpsMid-level Full TimeIndia14h ago
-
Software Development Specialist INR 940K-940KAmazon CloudWatch | Amazon DynamoDB | Amazon S3 | Amazon Web Services | Azure DevOpsMid-level Full TimeBangalore, KA, IN R14h ago
-
VP - Sr Lead Data Engineer (Snowflake) INR 4000K-6000KAgile | Data Governance | Data Modeling | Data Pipelines | Data QualityExecutive-level Full TimeMumbai, Maharashtra, India14h ago
-
Senior Manager INR 1500K-2800KAWS Glue | AWS Secrets | AWS Secrets Manager | Adaptive Query | Adaptive query executionHybrid work modeSenior-level Full TimePune, Maharashtra, India R15h ago
-
Senior Consultant - Data Engineering- INR 2000K-3500KApache Kafka | Apache Spark | Cloud Computing | Data Modeling | Data PipelinesSenior-level Full TimeMumbai, Maharashtra, India15h ago
-
Mid-level Full TimeBengaluru, Pune15h ago
-
Mid-level Full TimeIndia, Telangana, Hyderabad15h ago
-
Senior Data Engineer INR 3000K-4200KAPI Management | AWS | Application Architecture | Azure API | Azure API ManagementSenior-level Full TimeIndia15h ago
-
Senior-level Full TimeGurugram, Haryana, India16h ago
-
Apache Spark | Artificial Intelligence | Credit Risk | Cross-sell | Cross-sell analyticsSenior-level Full TimeMumbai, Maharashtra, India16h ago