AI Quality Engineer
Bengaluru, KA, India
INR 1800K-3000K (estimate) Senior-level Full Time
Tasks
- Build and maintain evaluation datasets with ground truth reference outputs and edge cases
- Build automated evaluation pipelines in CI CD
- Build evaluation frameworks for groundedness factuality relevance and task completion
- Build monitoring and alerting for production agent quality accuracy latency and error rates
- Conduct systematic prompt evaluation when prompts or models change
- Create test harnesses for LLM systems with mocks and fixtures
- Define quality metrics and thresholds for AI systems
- Define testing strategies for AI agents and agentic workflows
- Design behavioral test suites for non deterministic outputs
- Develop regression suites to detect quality degradation
- Diagnose quality failures and build regression tests
- Identify AI specific failure modes like hallucinations prompt injection drift and context limits
- Maintain test infrastructure and reporting dashboards
- Partner with engineers to define testability requirements during design
- Report quality metrics to stakeholders
- Review solution designs for quality and testability
- Track quality trends and trigger re evaluation when needed
- Translate acceptance criteria into test scenarios
Perks/Benefits
- N/A
Skills/Tech-stack
Application Insights | Azure | Behavioral Testing | CI/CD | Dataverse | Docker | Evaluation datasets | Git | GitHub | LLM Evaluation | Load Testing | Monitoring and Alerting | Prompt testing | Pytest | Python | RAG evaluation | REST APIs | Regression testing | SQL | TypeScript
Education
Bachelor of Engineering | Bachelor of Science | Master of Science | PhD
Roles
AI | AI Quality Engineer | Engineer | Quality Engineer | Software Engineer
Related jobs
-
Senior Staff AI Engineer INR 2500K-5000KAWS | Agents | Azure | Code Quality | Code Quality AutomationRelocation assistanceSenior-level Full TimeBengaluru, India8h ago
-
Senior-level Full Timebangalore, India12h ago
-
Senior GenAI Data Integration Engineer INR 3000K-5000KApp Engine | BigQuery | Cloud Functions | Cloud Run | Function CallingSenior-level Full TimeHyderabad, India15h ago
-
Senior-level Full TimeHyderabad, India15h ago
-
Senior-level Full TimeIN-HR-Gurgaon16h ago
-
Mid-level Full TimeBengaluru, Karnataka16h ago
-
Mid-level Full TimeBengaluru, Karnataka16h ago
-
Associate Engineer - Data Engineer pyTorch INR 980K-1500KData Pipelines | Data Transformation | Data cleaning | ETL | Feature EngineeringMid-level Full TimeBangalore - Karnataka, India - Kalyani …16h ago
-
Senior Manager INR 3000K-5000KFeature Engineering | Hugging Face | Hugging Face Transformers | Intent detection | Language ProcessingSenior-level Full TimeGurugram, Haryana, India17h ago
-
Senior-level Full TimeBangalore, Karnataka, India18h ago
-
Executive-level Full TimeBangalore, Karnataka, India18h ago
-
Senior Data Engineer (Python, PySpark, Kafka) INR 1500K-2500KApache Kafka | Data Warehousing | Dimensional Modeling | Git | GitHubSenior-level Full TimeBangalore, India19h ago
-
Mid-level Full TimeBengaluru, Karnataka, India20h ago
-
Scientist 4, Data Science INR 3700K-5000KAWS | Azure | Causal Inference | Deep learning | Distributed ComputingSenior-level Full TimeBengaluru, KA, India20h ago
-
Data Engineer INR 3000K-4000KAI machine learning | Cortex AI | Data Modeling | Machine Learning | Performance TuningSenior-level Full TimePune, Maharashtra, India20h ago
-
Engineer - Data Science INR 2500K-4000KAgile | Anomaly Detection | Azure IoT | Azure IoT Hub | Azure Machine LearningSenior-level Full TimeHadapsar, Pune, Maharashtra, IND, 41101320h ago
-
Sr. AI / ML Engineer – OpenAI Expert INR 2500K-5000KAI Search | Agent systems | Agentic AI | Agents SDK | Assistants APISenior-level Full TimePune, Maharashtra, India20h ago
-
Principal Platform Engineer-Databricks INR 3000K-4000KAWS | Automation | Azure | CI/CD | ComplianceCareer growth | Engineering culture focused on innovation | Global collaboration | Mission-driven workSenior-level Full TimeBangalore, Karnataka, IN21h ago
-
AI Analyst INR 180K-180KAI Foundry | Agent architecture | Autogen | Azure AI | Azure AI FoundryEmployee assistance program | Employee resource groups | Flexible time off | Medical insurance plans | Paid parental leaveEntry-level Full TimeCHANDIGARH, CHANDIGARH, India22h ago
-
Knowledge Graph, Graph DB, Python/Senior Consultant Specialist INR 3000K-4200KAgile methodology | Amazon Neptune | Data Governance | Data Lineage | Data ModelingContinuous professional development | Flexible working | Inclusive workplace | Opportunities for growthSenior-level Full TimeHyderabad, Telangana, India R22h ago
-
Senior Associate — Data Scientist, Applied AI/ML INR 1400K-2800KAWS | Agentic Workflows | Alerting | Azure | CI/CDMid-level Full TimeBengaluru, Karnataka, India23h ago
-
Lead Data Platform Engineer INR 3000K-4000KAWS Glue | AWS Lake Formation | AWS Lambda | AWS SageMaker | AWS SageMaker Unified StudioFamily wellbeing benefits | Health and wellbeing benefits | Hybrid work | Paid time offSenior-level Full TimeHyderabad, India23h ago
-
Data Scientist- Air Quality INR 1400K-1800KAirflow | Ansible | Apache Spark | CI/CD | Code reviewHealth and safety support | Hybrid work model | Mentoring | Professional developmentMid-level Full TimeNoida, Uttar Pradesh, India23h ago
-
Senior-level Full TimeBangalore, Karnataka, India23h ago
-
Data Engineer (AWS+Pyspark) INR 1500K-2000KAWS Glue | Amazon Athena | Amazon Aurora | Amazon DynamoDB | Amazon EC2Mid-level Full TimeHyderabad, Telangana, India23h ago