Lead Data Engineer
Tasks
- Build and govern Data Agent layer for hybrid AI workloads
- Build and maintain event and trigger pipelines from ERP to agent hub
- Define data classification and routing rules
- Design and build tenant registry and tenant data warehouse layer
- Design embeddings and vector database for RAG
- Drive automation and reliability for pipelines
- Establish observability and incident response practices
- Lead design and code reviews
- Mentor data engineers and provide technical escalation
- Optimize pipelines for low latency partitioning caching indexing
- Own data security access control lineage and governance
- Own end to end architecture for cloud data platform
- Set ETL ELT engineering standards
- Support ML ready platform data for predictive modeling
Perks/Benefits
Skills/Tech-stack
AWS | AWS Glue | Access Control | Airflow | Azure | BigQuery | DBT | Data Governance | Data Lineage | Data Quality | Data Security | Data Warehousing | Data orchestration | ELT | ETL | Embeddings | Flink | GCP | Grafana | Great Expectations | Java | Kafka | Kinesis | Multi-tenancy | NoSQL | Prometheus | Python | RAG | Redshift | SQL | Scala | Snowflake | Spark | Vector Databases
Education
Bachelor of Arts | Bachelor of Engineering | Bachelor of Science
Roles
Data Engineer | Engineer | Lead | Lead Data | Lead Data Engineer
Related jobs
-
AI Data Engineer III INR 3000K-4000KAWS | Agile methodologies | Amazon Redshift | Apache Airflow | CI/CDEducation assistance program | Fitness benefits | Flexible working hours | Health checkups | Hybrid work environmentSenior-level Full TimeBengaluru, Karnataka, India R3h ago
-
API Integration | Agent Frameworks | Agentic Workflows | Automation platforms | Event DrivenAutonomy | Fully remote | Healthcare benefits | Professional development | U S business hours alignmentMid-level Full TimeIndia R4h ago
-
API Development | AWS | Agents | Angular | AzureHome office setup budget | International multicultural environment | Learning and development budget | Visa and relocation support | Work from homeSenior-level Full TimeIndia R4h ago
-
Mid-level Full TimeIndia (Remote) R4h ago
-
Cloud ERP | Data Architecture | Data Catalog | Data Intelligence | Data QualityCollaborative & Inclusive Culture | Family support | Flexible working hours | Health and wellness benefits | Professional development opportunitiesSenior-level Full TimeIndia R11h ago
-
AI Agents | AI Governance | API Security | Auditability | Cloud SecurityHybrid work | Onsite supportMid-level Full TimeBengaluru, Karnataka, India R2d ago
-
Senior AI Engineer (m/f/d) INR 3242K-5000KAPIs | AWS | Agents | Angular | AzureHome office setup budget | Learning and development budget | Relocation support | Visa support | Work from homeSenior-level Full TimeRemote EMEA, Remote India R2d ago
-
BigQuery | CI/CD | Cloud Functions | Cloud Run | Cloud StorageContinuous professional development | Flexible workingMid-level Full TimeHyderabad, Telangana, India R2d ago
-
Analytics Engineer INR 2800K-4200KCI/CD | DBT | Dashboards | Data Quality | DatabricksHealth insurance | Hybrid work model | Mental health support | Paid time off | Transport allowanceSenior-level Full TimeBangalore (Hybrid) R2d ago
-
API | Apache Hadoop | Apache Kafka | Apache Spark | Azure DataHybrid workSenior-level Full TimeBangalore, Karnataka, India R3d ago
-
API | Agile | Apache Hadoop | Apache Kafka | Apache SparkHybrid work | Technical research opportunities | Training supportSenior-level Full TimeBangalore, Karnataka, India R3d ago
-
Apache Airflow | Apache Hadoop | Apache Kafka | Apache Spark | Azure DataHybrid work | Training opportunitiesSenior-level Full TimeBangalore, Karnataka, India R3d ago
-
Senior Software Engineer II (Java Microservices) INR 3000K-4500KAWS | Agile | Apache Kafka | Apache Spark | AzureCareer development | Distributed collaboration | Learning opportunities | Work from Anywhere cultureSenior-level Full TimeIndia R3d ago
-
Senior-level Full TimePune, MH, IN R3d ago
-
Senior Data Analyst INR 2817K-4000KAlation | Analysis Services | Azure Analysis | Azure Analysis Services | Azure DataSenior-level Full TimeTelangana, India R3d ago
-
AWS | Ansible | As-a-Service | Azure | BIOSSenior-level Contract Full TimeIndia - Remote R3d ago
-
Data Science & Engineering Lead INR 3000K-4500KAPI Gateway | AWS Glue | AWS Lambda | Airbyte | Amazon EC2Senior-level Full TimeIndia - Remote R3d ago
-
AI Agents | AI Data | AI data governance | API Security | Cloud SecurityMid-level Full TimeBengaluru, Karnataka, India R3d ago
-
AWS | AWS Glue | Agile | Amazon S3 | CI/CDSenior-level Full TimeHyderabad, Telangana, India - Remote R3d ago
-
Senior AI and Machine Learning Engineer INR 3000K-4680KConvolutional Neural Networks | Data Mining | Data Preprocessing | Data Visualization | Decision TreesCareer Development Programs | Flexible work arrangements | Health and wellbeing benefits | Hybrid work schedule | Inclusion initiativesSenior-level Full TimeBengaluru, Karnātaka, India R3d ago
-
A/B | A/B Testing | AWS | Apache Spark | AzureFlexibility programmes | Hybrid work | Inclusive benefits | MentorshipSenior-level Full TimeBengaluru Millenia, India R3d ago
-
A/B | A/B Testing | AIOps | AWS | Artificial IntelligenceFlexibility programmes | Hybrid work environment | MentorshipSenior-level Full TimeBengaluru Millenia, India R3d ago
-
Senior AI and Machine Learning Engineer INR 3000K-4680KConvolutional Neural Networks | Data Mining | Data Modeling | Data Visualization | Decision TreesCareer Development Programs | Flexible work arrangement | Health and wellbeing benefitsSenior-level Full TimeBengaluru, Karnātaka, India R3d ago
-
Senior AI Engineer INR 2500K-5000KAPI Management | AWS | Agile | Application Programming | Application Programming InterfacesSenior-level Full TimeKolkata, West Bengal, India R3d ago
-
Developer, Java,Spark and GCP INR 2771K-4064KApache Hadoop | Apache Spark | Cloud platform | GCP | Google CloudHybrid work | Work-life flexibilitySenior-level Full TimeChennai, India R3d ago