Lead, Data Engineering
IN - HYDERABAD ORION, India
INR 2771K-4000K (estimate) Senior-level Full Time
Tasks
- Automate deployments using CI/CD and Infrastructure-as-Code
- Design tables and data publication patterns
- Enable auditability and lineage controls
- Implement ingestion transformation and monitoring best practices
- Integrate pipelines with data mastering platform
- Maintain deployment monitoring and incident management procedures
- Maintain documentation and knowledge sharing
- Maintain secure compliant platform operations
- Manage secure data onboarding access permissions
- Migrate data pipelines to Databricks
- Monitor and observe production data pipelines
- Onboard new data assets to enterprise platform
- Operate and troubleshoot data pipelines
- Optimize pipeline performance
- Perform data quality checks metadata management reconciliation
- Provide technical guidance mentor engineers
- Review code and maintain quality
- Standardize Bronze Silver and Gold data layers
- Support interoperability across AWS and Databricks
- Support metadata driven onboarding workflows
- Support production support issue resolution performance tuning
- Troubleshoot ingestion and mastering workflow issues
- Validate and publish data assets
Perks/Benefits
- N/A
Skills/Tech-stack
AWS Glue | AWS Glue Data Catalog | AWS IAM | AWS Lake | AWS Lake Formation | AWS Lambda | AWS Step Functions | Amazon Athena | Amazon CloudWatch | Amazon Kinesis | Amazon S3 | Apache Iceberg | Automated testing | CI/CD | Data Catalog | Data Governance | Data Lineage | Data Quality | Databricks | Delta Lake | Glue Data Catalog | Incident Management | Infrastructure as Code | Lake Formation | Metadata Management | Monitoring | PySpark | Python | Release Management | SQL | Spark | Step Functions | Unity Catalog | Version control | “as-code”
Education
N/A
Roles
Data Engineer | Data Platform | Data Platform Lead | Engineer | Lead | Platform Lead | Senior Data Engineer
Related jobs
-
Senior Software Engineer (Universal Robots, Bangalore - India) INR 2475K-3500KAgile | C++ | CI/CD | Clean Code | Code reviewExposure to Cutting Edge Robotics Technology | Flexible working conditions | Health insurance | Inclusive work environment | Professional development opportunitiesSenior-level Full TimeBangalore, IN6h ago
-
Senior Software Engineer, Machine Learning INR 3264K-5400KA/B | A/B Testing | Aerospike | B testing | Bayesian analysisCommuter benefits | Dental insurance | Disability insurance | Financial wellness support | Flexible remote workSenior-level Full TimeBengaluru, India9h ago
-
Senior Software Engineer (Data Platform) INR 2475K-3500KAWS | Airflow | Apache Flink | Apache Hadoop | Apache HiveCommuter benefits | Dental insurance | Disability insurance | Financial wellness support | Healthcare benefitsSenior-level Full TimeBengaluru, India9h ago
-
Senior Cloud Storage Engineer – NetApp CVO on GCP, AVP INR 2755K-3500KAutomation | CI/CD | Cloud Security | Cloud platform | Cloud volumes ONTAPAccident insurance | Certification sponsorship | Childcare assistance reimbursement | Employee assistance program | Gender neutral parental leaveSenior-level Full TimePune - Business Bay, India16h ago
-
Mid-level Full TimePune, India20h ago
-
Senior Software Engineer INR 2000K-2700KAgile | Cloud services | Code review | Data Modeling | Data WarehousingSenior-level Full TimePune, India20h ago
-
Artificial intelligence Developer (Pro Code) INR 2500K-4680KAI Foundry | APIs | Asynchronous processing | Autogen | Azure AISenior-level Full TimeThane, Maharashtra, India21h ago
-
Manager Data and Analytics INR 1400K-2000KAmazon Redshift | Azure | BigQuery | Business Intelligence | Cloud platformMid-level Full TimeBangalore, Karnataka, India R22h ago
-
Senior Artificial Intelligence/Machine Learning Engineer INR 2500K-5000KAPI Integration | AWS SageMaker | Agent systems | Azure ML | CCPACompany paid medical insurance | Company-paid certifications | Financial and legal consultations | Internal workshops and meetups | Language coursesSenior-level Full TimeIndia23h ago
-
Senior AI Engineer INR 3242K-4680KAWS EC2 | AWS ECS | AWS Lambda | AWS RDS | Amazon S3Financial consultation | Legal consultation | Medical insurance | Mental health support | Paid certificationsSenior-level Full TimeIndia23h ago
-
Associate Engineer - AI/ML with Python INR 2500K-4000KAWS | Bitbucket | CI/CD | CSS | CeleryCollaborative work environment | Onsite work 5 days a week | Training and developmentMid-level Full TimeBangalore - Karnataka, India - Kalyani …1d ago
-
Data Scientist – AI & Cybersecurity INR 2500K-4500KAPIs | AWS | Adversarial Testing | Amazon SageMaker | Anomaly DetectionGrowth opportunities | Mentorship programs | Training opportunitiesSenior-level Full TimeBangalore - Karnataka - India1d ago
-
Senior Python GenAI Engineer INR 2000K-4580KAWS | Angular | Anthropic | Azure | Azure OpenAIEmployee Assistance Program (EAP) | Flexible working environment | LinkedIn Learning | Volunteer time offSenior-level Full TimePune, MH, India1d ago
-
Senior Java Kafka Architect INR 2000K-2700KAmazon Web Services | Apache Kafka | CI/CD | Code Refactoring | Code reviewSenior-level Full TimeIN-AP-Hyderabad1d ago
-
Senior-level Full TimeIN-TN-Chennai1d ago
-
Data Engineer INR 2000K-3000KAPI Design | Asynchronous programming | Caching | Celery | Clean architectureSenior-level Full TimeIN-KA-Bangalore1d ago
-
Azure Data Engineer INR 1500K-2500KApache Spark | Azure Data | Azure Data Lake | Azure Databricks | Data GovernanceMid-level Full TimeIN-KA-Bangalore1d ago
-
Mid-level Full TimeIN-KA-Bangalore1d ago
-
Software Engineer III, ML Infrastructure, Google Cloud INR 2500K-5000KBack-end | Back-end development | C++ | Cloud APIs | Data StorageSenior-level Full TimePune, Maharashtra, India1d ago
-
Campus FTE INR 1200K-2500KAgentic AI | Amazon Web Services | Classification | Cloud platform | Data AnalysisMid-level Full TimeBangalore, Karnataka, India1d ago
-
AI Engineer INR 2000K-5000KAWS SageMaker | Azure Machine Learning | C++ | Computer Vision | Deep learningSenior-level Full TimeIndia - Pune1d ago
-
Campus FTE INR 1800K-3000KAgentic AI | Amazon Web Services | Classification | Cloud platform | Data QualityMid-level Full TimeGurgaon, Haryana, India1d ago
-
Mid-level Full TimeIndia, Telangana, Hyderabad; India, Karnataka, Bangalore; …1d ago
-
Senior Data Platform Consultant INR 3000K-4000KAWS architecture | Agile | Amazon Web Services | Azure | Azure ArchitectureHybrid work model | Mentoring and career development | Performance based learning and enablement sessionsSenior-level Full TimePune, India1d ago
-
Associate TSE II PHP 3000KAWS | Active Directory | Authentication | Azure | BenchmarkingHybrid workMid-level Full TimeBengaluru1d ago