Data Engineer - Generative AI & Vector Systems
Chennai, Tamil Nadu, India
INR 2500K-5000K (estimate) Senior-level Full Time
Tasks
- Automate workflow orchestration using Apache Airflow
- Build RAG pipelines
- Build batch and near real time ingestion pipelines
- Build pipelines to clean transform chunk enrich and load data into vector databases
- Design and manage vector database architectures
- Design efficient data models for AI workloads
- Design scalable data ingestion pipelines for AI and ML
- Develop cloud-native solutions on AWS
- Develop production grade ETL and ELT pipelines
- Ensure data integrity and governance
- Generate embeddings from structured and unstructured data
- Implement monitoring, logging, and alerting
- Improve similarity search performance and reduce retrieval latency
- Manage embedding generation and vector indexing
- Monitor pipeline performance and ensure high availability
- Optimize data quality for semantic search and retrieval
- Optimize indexing storage metadata management and retrieval performance
- Troubleshoot production data issues
- Use Starburst or Trino to federate and query multiple data platforms
- Write complex SQL queries across distributed data sources
Perks/Benefits
Skills/Tech-stack
AWS Glue | AWS IAM | AWS Lambda | Amazon Athena | Amazon CloudWatch | Amazon EMR | Amazon S3 | Apache Airflow | Chroma | Data Cleansing | Data Modeling | Data Transformation | ELT | ETL | Embeddings | Faiss | JSON | Metadata Management | Milvus | Pinecone | Prompt engineering | Python | Qdrant | REST API | Retrieval-Augmented Generation | SQL | Semantic Search | Starburst | Trino | Vector Databases | Vector Search | Weaviate
Education
N/A
Roles
Related jobs
-
Java Fullstack Engineer with GenAI & React INR 1500K-2500KAgile Development | Azure | Design Patterns | Docker | Extreme programmingEmployee assistance program | Flexible working environment | LinkedIn Learning | Volunteer time offMid-level Full TimeChennai, TN, India7h ago
-
Senior-level Full TimeMumbai, Maharashtra, India8h ago
-
Embedded Hardware Development INR 2500K-4460KCAN-FD | Codebeamer | DSO | EMMC | Engineering Test ManagementSenior-level Full Timebengaluru, India8h ago
-
Software Engineer INR 1500K-2100KAWS | Agile | Bash | CI/CD | CloudWatchFamily wellbeing benefits | Health and wellbeing benefits | Hybrid work | Paid time off | Work-life balance benefitsMid-level Full TimeHyderabad, India R9h ago
-
ML Engineer INR 2000K-3500KAutomated Pipelines | C# | C++ | Classification | ClusteringAnnual leave | Employee assistance programme | Employee wellbeing platform | Flexible benefits | Flexible workingSenior-level Full TimeBangalore, IN10h ago
-
Staff Engineer - Data Path INR 1500K-3000KC# | C++ | Cache Management | Concurrency | Data transferSenior-level Full TimePune Office10h ago
-
Control Path Engineer INR 2000K-3000KAPI Design | Automation | C++ | CI/CD | Configuration ManagementSenior-level Full TimePune Office11h ago
-
AWS | Artificial Intelligence | Azure | CI/CD | Computer VisionCareer development | Inclusive culture | Professional learning | Remote workSenior-level Full TimeIndia R13h ago
-
Data / Retrieval Engineer INR 3200K-4720KAPIs | Access Control | Attribute-Based Access Control | Attribute-based access | ChunkingSenior-level Full TimeNoida/Gurgaon, UP, IN R15h ago
-
AI Orchestration / Platform Engineer INR 2500K-4680KAlerting | Authentication | Authorization | CI/CD | CachingSenior-level Full TimeNoida/Gurgaon, HR, IN R15h ago
-
Tech S And T-Cloud AI Platform Engineer-ISR-Senior-GDSF02 INR 3000K-5000KAI Pipelines | ARM Templates | AWS | AWS CloudFormation | Amazon BedrockSenior-level Full TimeNoida, UP, IN, 20130115h ago
-
Sr Analytics Engineer INR 2400K-3000KAPI Integration | AWS | Amazon Redshift | Data Analysis | Data GovernanceRelocation assistanceSenior-level Full TimeBengaluru, India15h ago
-
Principal Agentic AI Engineer / Hands-on Technical Lead INR 3600K-5000KAI tracing | Agent patterns | Authentication | Authorization | AutogenSenior-level Full TimeNoida/Gurgaon, HR, IN R15h ago
-
Machine Learning Engineer 3 ( Firefly ) INR 3264K-5000KCUDA | Diffusion Models | Distributed Systems | GANs | GPU resource managementSenior-level Full TimeNoida, India R15h ago
-
AI Services | Azure AI | Azure AI Services | Azure OpenAI | Azure SpeechHybrid work | Onsite work | Travel 10 percentSenior-level Full TimeKarnataka, Bengaluru Old No 62, Plot … R15h ago
-
Senior Data Engineer, Director INR 1500K-3000KAWS EMR | Amazon Athena | Amazon DynamoDB | Amazon Kinesis | Amazon QuickSightSenior-level Full TimeGurugram, India15h ago
-
Architect - Emerging Technologies Engineer (Python, AI) INR 2000K-5000K.NET | AI Search | AWS Bedrock | Anthropic API | AutogenSenior-level Full TimeHyderabad, India15h ago
-
Sr Engineer - AI INR 2500K-4800KAgentic AI | Amazon S3 | Anthropic | Context engineering | ElasticsearchSenior-level Full TimeTower 02, Manyata Embassy Business Park, …15h ago
-
ARM | ASIC | Arc | Bitbucket | C#Entry-level Full TimeBengaluru, KA, India1d ago
-
Mid-level Full TimeBangalore, Karnataka, India1d ago
-
Analyst/Consultant - Data Engineer - Azure (INDM) INR 1500K-2000KApache Cassandra | Azure Blob | Azure Blob Storage | Azure Data | Azure Data FactoryMid-level Full TimePune, Maharashtra, India1d ago
-
Staff AI Quality Engineer INR 2000K-5000KAWS Glue | AWS Lambda | AWS SageMaker | Amazon Athena | Amazon BedrockSenior-level Full TimeBangalore, KA, India1d ago
-
Senior-level Full TimeBangalore, KA, India1d ago
-
Senior Database Engineer INR 2000K-2500KBackup and Recovery | Cassandra | Data Modeling | Data Warehousing | Database DesignSenior-level Full TimeBengaluru, Karnataka, India1d ago
-
AI Foundry | API Design | Agent Framework | Agents | AzureAI-native team | Enterprise product impact | Onsite work | Ownership of production systemsMid-level Full TimeNOIDA, Uttar Pradesh, India1d ago