Lead Data Engineer - AI Platforms
Primary location: Bengaluru, Karnataka
INR 2500K-4500K (estimate) Senior-level Full Time
Tasks
- Architect AI first enterprise data platforms
- Build AI ready foundations for feature stores embeddings and vector datasets
- Create resilient fault tolerant data pipelines with idempotency retry checkpointing and recovery
- Define enterprise data strategy and architecture standards
- Design distributed data systems with partitioning sharding clustering and parallel processing
- Design end to end data ingestion processing storage serving and governance
- Design semantic layers and knowledge architectures including semantic APIs and knowledge services
- Enforce compliance for HIPAA GDPR and PII protection
- Establish data governance frameworks including metadata catalog lineage and traceability
- Implement data observability with monitoring logging lineage tracking and alerting
- Implement data quality controls anomaly detection deduplication and SLA monitoring
- Implement knowledge graphs ontologies semantic search and vector search
- Implement lakehouse and data mesh based architectures
- Implement lambda and kappa batch streaming and real time processing
- Implement security controls including RBAC encryption data masking and access auditing
- Lead platform modernization cloud transformation and AI data platform initiatives
- Mentor engineering teams and drive technical excellence
- Optimize storage indexing partitioning query performance and data lifecycle management
Perks/Benefits
Skills/Tech-stack
Access auditing | Agentic AI | Amazon Redshift | Apache Spark | Azure Synapse | Azure Synapse Analytics | BigQuery | Data Governance | Data Lake | Data Lineage | Data Observability | Data Quality | Data Warehouse | Data masking | Data mesh | Databricks | Distributed Systems | Embeddings | Encryption | Fault Tolerance | Feature Store | GDPR | HIPAA | Kafka | Kappa Architecture | Knowledge graphs | Lakehouse | Lambda Architecture | Medallion Architecture | Metadata Management | Ontologies | PII | PySpark | Python | RAG | RBAC | Retrieval-Augmented Generation | SQL | Scalability | Semantic Search | Semantic layers | Snowflake | Synapse Analytics | Vector Databases | Vector Search
Education
Bachelor of Engineering | Bachelor of Science | Master of Science
Roles
Data Engineer | Engineer | Lead | Lead Data | Lead Data Engineer
Related jobs
-
Manager Software Engineer- ML + NLP+ ppl management INR 3200K-4800KAPIs | BERT | Data Mining | Data Science | Deep learningFlexible work arrangements | Hybrid work environmentSenior-level Full TimeGurgaon - Cyber Park, India3h ago
-
Principal Engineer, Storage Data Science and Analytics INR 3200K-4590KAIOps | Anomaly Detection | Apache Spark | C# | C++Career Development Programs | Flexible work arrangements | Health and wellbeing benefitsSenior-level Full TimeBengaluru, Karnātaka, India3h ago
-
AWS | Airflow | DBT | Fine Tuning | Language ModelsBonuses | Disability insurance | Life insurance | Paid parental leave | Paid time offSenior-level Full TimeRemote, India R9h ago
-
AI/ML engineer - Associate Research Scientist INR 1500K-2500KAIOps | API Development | AWS | Anomaly Detection | AutomationContinuous learning and professional development | Employee Assistance Program (EAP) | Flexible working environment | LinkedIn Learning | Volunteer time offMid-level Full TimeChennai, TN, India10h ago
-
Mid-level Full TimeHadapsar, Pune, Maharashtra, IND, 41101310h ago
-
Efs Fcrm 169474 INR 1500K-2400KApache Spark | Azure DevOps | CI/CD | Data Partitioning | Data WarehousingMid-level Full TimeHyderabad, IND-TG, India10h ago
-
Lead Data Analyst – Business & Process INR 1800K-2700KAnalytics Platforms | Budgeting | CRM | Dashboarding | DashboardsCareer growth support | Joining support | Learning and development opportunities | Life Coverage | Maternity and paternity benefitsExecutive-level Full TimeHyderabad, TS, India12h ago
-
Senior Database Engineer - #3190 (Remote) INR 1500K-2700KAWS IAM | AWS KMS | AWS RDS | Amazon Aurora | BashOn-call rotation | Remote workSenior-level Full TimeChennai, India R13h ago
-
Software Engineer SRE (Site Reliability Engineer) INR 1600K-3000KAWS | Ansible | Apache Airflow | Apache Kafka | Apache SparkSenior-level Full TimeBangalore, India Office (BANGALORE) R14h ago
-
Knowledge Graph Engineering Lead INR 3500K-5000KAgile | Amazon Neptune | CI/CD | Data Governance | Data LineageFlexible working | Inclusive work environment | Professional developmentSenior-level Full TimeHyderabad, Telangana, India R14h ago
-
Mid-level Full TimeIN-TN-Chennai15h ago
-
Senior-level Full TimeIN-MH-Pune15h ago
-
Snr AI INR 2500K-5000KAI Search | Amazon SageMaker | Azure AI | Azure AI Search | Azure Machine LearningSenior-level Full TimeIN-HR-Gurgaon15h ago
-
Senior-level Full TimeBengaluru, Karnataka15h ago
-
Deputy Manager - Fraud Analytics & Monitoring INR 800K-1250KData Science | Data analytics | Excel | Machine Learning | PythonAnnual health checkup | Dental benefits | Group term life insurance | Inclusive health benefits | Learning and development programMid-level Full TimeGurugram, India15h ago
-
GCP BigQuery,DevOps,Credit Card Domain/Associate Director, Data and Analytics Specialist INR 650K-900KApache Airflow | Apache Kafka | BigQuery | CI/CD | Cloud ComposerMid-level Full TimeHyderabad, Telangana, India R16h ago
-
Azure Data Engineer INR 1500K-2000KAzure Data | Azure Data Factory | Azure Data Lake | Azure Functions | Azure SynapseSenior-level Full TimePune, Maharashtra, India16h ago
-
Lead Software Engineer - Python Full Stack, AI/ML INR 4000K-5000KAWS | Agile | Angular | Application Resiliency | Automated testingSenior-level Full TimeMumbai, Maharashtra, India16h ago
-
Senior-level Full Timecoimbatore, India17h ago
-
Digital Marketing Analyst INR 2500K-3380KAWS | Azure | Cloud platform | Customer Analytics | Data GovernanceSenior-level Full TimeChennai, Tamil Nadu, India17h ago
-
Senior Staff Machine Learning Engineer (Java + LLM) INR 3000K-4401KAlgorithm Design | Code Tuning | Data Structures | Data Structures and Algorithms | Design PatternsFlexible work schedule | Health benefits | Paid time off | Remote work optionSenior-level Full TimeBangalore, Karnataka, India17h ago
-
Senior Staff Engineer, Data Migration Lead INR 3000K-4000KAccess Control | Audit Trail | Cutover planning | Data Governance | Data ManagementSenior-level Full TimeRemote, India R18h ago
-
Senior-level Full TimeMumbai, Maharashtra, India18h ago
-
Data Analyst Customer Success Engineer INR 2500K-4064KA/B | A/B Testing | AI workflows | API Integration | AmplitudeClient Customer Interactions | Health insurance | Learning aids | Meal vouchersSenior-level Full TimeHYDERABAD, Telangana, India18h ago
-
AI Platform Engineer INR 3700K-5500KAPI Design | Agent Orchestration | Agent systems | Artificial Intelligence | CI/CDSenior-level Full TimeMumbai, India19h ago