Data Engineer - Generative AI & Vector Systems
Chennai, Tamil Nadu, India
INR 2500K-5000K (estimate) Senior-level Full Time
Tasks
- Automate workflow orchestration using Apache Airflow
- Build RAG pipelines
- Build batch and near real time ingestion pipelines
- Build pipelines to clean transform chunk enrich and load data into vector databases
- Design and manage vector database architectures
- Design efficient data models for AI workloads
- Design scalable data ingestion pipelines for AI and ML
- Develop cloud-native solutions on AWS
- Develop production grade ETL and ELT pipelines
- Ensure data integrity and governance
- Generate embeddings from structured and unstructured data
- Implement monitoring, logging, and alerting
- Improve similarity search performance and reduce retrieval latency
- Manage embedding generation and vector indexing
- Monitor pipeline performance and ensure high availability
- Optimize data quality for semantic search and retrieval
- Optimize indexing storage metadata management and retrieval performance
- Troubleshoot production data issues
- Use Starburst or Trino to federate and query multiple data platforms
- Write complex SQL queries across distributed data sources
Perks/Benefits
Skills/Tech-stack
AWS Glue | AWS IAM | AWS Lambda | Amazon Athena | Amazon CloudWatch | Amazon EMR | Amazon S3 | Apache Airflow | Chroma | Data Cleansing | Data Modeling | Data Transformation | ELT | ETL | Embeddings | Faiss | JSON | Metadata Management | Milvus | Pinecone | Prompt engineering | Python | Qdrant | REST API | Retrieval-Augmented Generation | SQL | Semantic Search | Starburst | Trino | Vector Databases | Vector Search | Weaviate
Education
N/A
Roles
Related jobs
-
Database Engineer INR 500K-1500KCI/CD | DB2 | Data Warehousing | Database performance | Database performance tuningMid-level Full TimePune, MH, India5h ago
-
Senior AI Engineer I INR 3264K-5876KArtificial Intelligence | BigQuery | Cloud Composer | Cloud platform | Data ProcessingSenior-level Full TimeBengaluru, KA, India5h ago
-
Principal Engineer, Data Analytics Engineering [ 8+ years ] INR 2500K-3584KAWS | Algorithms | Azure | Big Data | CI/CDSenior-level Full TimeBengaluru, KA, India6h ago
-
Agent Development | Agent Development Kit | Angular | Backend Development | C++Mid-level Full TimeBengaluru, Karnataka, India6h ago
-
Lead Software Engineer - Java and AI/ML INR 3500K-5000KAI Agents | Agile | Automated testing | CI/CD | Code modernizationSenior-level Full TimeMumbai, Maharashtra, India8h ago
-
AI Engineer I INR 2500K-5000KAWS | Azure | Azure OpenAI | Docker | FastAPIWork from office 5 days per weekSenior-level Full TimeMumbai, India8h ago
-
Mid-level Full TimePune, Maharashtra, India9h ago
-
Databricks Developer INR 1500K-2400KACID | Batch Processing | CI/CD | Cluster management | Data IngestionWork from office 5 days per weekMid-level Full TimeGurugram, Haryana, India9h ago
-
Associate-Digital Product Management INR 1685K-3300KAPI Integration | Anthropic | Data Governance | Data Ingestion | Data ValidationCareer growth opportunities | Holistic well-being support | Leadership development programs | Training opportunitiesMid-level Full TimeGurugram, HR, India10h ago
-
Associate-Digital Product Management INR 1685K-3300KAPIs | Anthropic | Data Quality | Data Visualization | Data integrationMid-level Full TimeGurugram, HR, India10h ago
-
Principal Data Scientist INR 3000K-4000KContainers | Continuous integration | Decoder Architecture | Deep learning | Design PatternsCar lease | Certification programs | Continuous learning programs | Corporate pension plan | Dental coverageSenior-level Full TimeBangalore, Karnataka, India10h ago
-
Lead Assistant Manager - Azure Data Engineer INR 1500K-2500KAgile | Amazon Redshift | Apache Airflow | Apache Hadoop | Apache HiveSenior-level Full TimeNoida, Uttar Pradesh, India10h ago
-
Executive - Data Governance INR 600K-840KCI/CD | Data Cleansing | Data Denormalization | Data Modeling | Data NormalizationExecutive-level Full TimeBangalore, Karnataka, India10h ago
-
Sr Data Engineer, Python + Spark (Data Federation skillset - Data Lakehouse - Eg: Starburst) - Chennai INR 1500K-2500KAWS EMR | AWS Glue | AWS S3 | Apache Hudi | Apache Iceberg401k retirement plan | Dental insurance | Medical insurance | Paid Holidays | Paid time offSenior-level Full TimeIndia11h ago
-
Lead Software Engineer - Machine Learning INR 3000K-5000KAPI Microservices | Agent systems | Artificial Intelligence | Asynchronous Workflows | Distributed SystemsSenior-level Full TimeBengaluru, India11h ago
-
Assistant Manager- Python Gen AI Engineer INR 1200K-2500KAI Agents | API Development | Agile | Containerization | Data PipelinesWork from client office 5 days per weekMid-level Full TimeBangalore, Karnataka, India11h ago
-
Cloud DevOps - Consultant INR 1500K-3000KAWS | Apache Airflow | Apache Kafka | Apache Spark | AzureMid-level Full TimeGurgaon, Haryana, India11h ago
-
Senior Consultant_Python Engineer - Senior INR 1500K-3465KAWS Glue | Agile | Amazon Web Services | Apache Airflow | CI/CDSenior-level Full TimeThane, IN, 40060412h ago
-
Senior Lead Software Engineer - Java/Python, AI INR 3500K-5500KApplication Programming | Application Programming Interfaces | Authentication | Authorization | Automated testingSenior-level Full TimeMumbai, Maharashtra, India13h ago
-
Manager - Azure Data Engineer INR 1500K-2500KAgile | Amazon Redshift | Apache Airflow | Apache Hadoop | Apache HiveMid-level Full TimeNoida, Uttar Pradesh, India13h ago
-
Power BI + Python INR 500K-1500KAzure Data | Azure Data Factory | Azure SQL | Azure Synapse | CI/CDMid-level Full TimeTelangana, India13h ago
-
Entry-level Full TimeBangalore, Karnataka, India13h ago
-
Entry-level Full TimeBangalore, Karnataka, India13h ago
-
Analyst-KDNI INR 300K-360KAPI documentation | Amazon Neptune | Amazon Web Services | ArgoCD | AutogenEntry-level Full TimeBangalore, Karnataka, India13h ago
-
Analyst-KDNI INR 300K-360KAgent systems | Amazon Neptune | ArgoCD | Artificial Intelligence | AutogenEntry-level Full TimeBangalore, Karnataka, India13h ago