AI Quality Engineer
Bengaluru, KA, India
INR 1800K-3000K (estimate) Senior-level Full Time
Tasks
- Build and maintain evaluation datasets with ground truth reference outputs and edge cases
- Build automated evaluation pipelines in CI CD
- Build evaluation frameworks for groundedness factuality relevance and task completion
- Build monitoring and alerting for production agent quality accuracy latency and error rates
- Conduct systematic prompt evaluation when prompts or models change
- Create test harnesses for LLM systems with mocks and fixtures
- Define quality metrics and thresholds for AI systems
- Define testing strategies for AI agents and agentic workflows
- Design behavioral test suites for non deterministic outputs
- Develop regression suites to detect quality degradation
- Diagnose quality failures and build regression tests
- Identify AI specific failure modes like hallucinations prompt injection drift and context limits
- Maintain test infrastructure and reporting dashboards
- Partner with engineers to define testability requirements during design
- Report quality metrics to stakeholders
- Review solution designs for quality and testability
- Track quality trends and trigger re evaluation when needed
- Translate acceptance criteria into test scenarios
Perks/Benefits
- N/A
Skills/Tech-stack
Application Insights | Azure | Behavioral Testing | CI/CD | Dataverse | Docker | Evaluation datasets | Git | GitHub | LLM Evaluation | Load Testing | Monitoring and Alerting | Prompt testing | Pytest | Python | RAG evaluation | REST APIs | Regression testing | SQL | TypeScript
Education
Bachelor of Engineering | Bachelor of Science | Master of Science | PhD
Roles
AI | AI Quality Engineer | Engineer | Quality Engineer | Software Engineer
Related jobs
-
Data Bricks Engineer INR 1500K-2400KDatabricks | Databricks SQL | Delta Lake | JSON-RPC | MLflowWork from office 5 days per weekSenior-level Full TimeGurugram, Haryana, India1h ago
-
Staff Engineer ( Meshing, C++ / C#, GenAI ) INR 1753K-2800KC# | C++ | CUDA | Computational Geometry | DebuggingHealth insurance | Wellness benefitsSenior-level Full TimeIndia Off-Site, India1h ago
-
Mid-level Full TimeChennai, Tamil Nadu, India2h ago
-
Senior AI Engineer INR 3000K-5000KAPI deployment | Adversarial Testing | Auto Retraining | Azure ML | CI/CDSenior-level Full TimeBangalore8h ago
-
Azure Data | Azure Data Factory | CI/CD | Data Factory | Data ReconciliationSenior-level Full TimeBangalore, India8h ago
-
Senior Talend Developer / Lead Data Engineer INR 1500K-2700KAzure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure DevOpsSenior-level Full Timebengaluru, India9h ago
-
Solutions Architect INR 2400K-3500KADLS | AWS | Airflow | Azure | CassandraBootcamp training | Broadband reimbursement | Continuous learning opportunities | Medical insurance | Paid certificationsSenior-level Full TimeIndia - Remote R9h ago
-
Sr. SW Engineer, Machine Learning INR 2475K-3465KA/B | A/B Testing | Agent systems | B testing | CI/CDFinancial wellness support | Hybrid work | Mental health support | Time offSenior-level Full TimeBengaluru, India9h ago
-
Sr GIS Data Engineer -Python l AI ML l SQL INR 1800K-2500KArcGIS | Batch Processing | Data Pipelines | Data Quality | FMESenior-level Full TimeMumbai, MH, India10h ago
-
Mid-level Full TimeIT Centre Bengaluru GDTC, India14h ago
-
Senior Data Engineer INR 1500K-2500KAccess Control | Azure | Azure Data | Azure Data Factory | Azure SQLSenior-level Full TimeHyderabad, IN, 500 00814h ago
-
Java Technical Lead- Career INR 2000K-3000KAWS | Airflow | Apache Beam | Atlassian | AzureHealthcare package | Hybrid work | Learning platform with guided career tracks | Paid time offSenior-level Full TimeIND-Trivandrum-Equifax Analytics-PEC, India14h ago
-
IN_Senior Associate_Data Analytics / Data Engineering_Digital Integration_Advisory_Gurgaon INR 2000K-2600K.NET | Agile | Azure | Azure Databricks | Azure SQLSenior-level Full TimeGurugram 10 C, India14h ago
-
IN_Senior Associate_Data Analytics / Data Engineering_Digital Integration_Advisory_Gurgaon INR 2000K-2600K.NET | Agile | Azure | Azure Databricks | C#Senior-level Full TimeGurugram 10 C, India14h ago
-
IN_Senior Associate_Data Analytics / Data Engineering_Digital Integration_Advisory_Gurgaon INR 2000K-2600K.NET | Agile | Azure | Azure Databricks | C#Flexibility programmes | Inclusive benefits | Mentorship | Wellbeing supportSenior-level Full TimeGurugram 10 C, India14h ago
-
IN_Senior Associate_Data Analytics / Data Engineering_Digital Integration_Advisory_Gurgaon INR 2000K-2600K.NET | Agile | Azure | Azure Databricks | C#Flexibility programs | Inclusive benefits | MentorshipSenior-level Full TimeGurugram 10 C, India14h ago
-
IN_Senior Associate_Azure Data Engineer_GCC_Advisory_Bangalore INR 2000K-2800KAI/ML | Azure Data | Azure Data Factory | Azure Databricks | Azure SQLSenior-level Full TimeBengaluru Millenia, India14h ago
-
IN_Senior Associate_Azure Data Engineer_GCC_Advisory_Bangalore INR 2000K-2800KAzure Data | Azure Data Factory | Azure Databricks | Azure SQL | Azure SQL DatabaseFlexibility programmes | Mentorship | Wellbeing supportSenior-level Full TimeBengaluru Millenia, India14h ago
-
IN_Senior Associate_Azure Data Engineer_GCC_Advisory_Bangalore INR 2000K-2800KAI/ML | AI/ML data | AI/ML data preparation | Azure Data | Azure Data FactorySenior-level Full TimeBengaluru Millenia, India14h ago
-
IN_Senior Associate_Azure Data Engineer_GCC_Advisory_Bangalore INR 2000K-2800KAzure Data | Azure Data Factory | Azure Databricks | Azure SQL | Azure SQL DatabaseEqual employment opportunities | Flexibility programs | Inclusive benefits | Mentorship | Wellbeing supportSenior-level Full TimeBengaluru Millenia, India14h ago
-
Mid-level Full TimeHyderabad, Telangana, India14h ago
-
EY - GDS Consulting - AIA -Generative AI - Senior INR 1960K-3300KAWS CloudFormation | Amazon Web Services | Apache Spark | Azure | Azure DevOpsCoaching and Mentorship | Continuous learning | Diverse and inclusive culture | Tools and flexibilitySenior-level Full TimeKolkata, WB, IN, 70009114h ago
-
Senior-level Full TimeBangalore, India14h ago
-
Senior-level Full TimeIndia - Bangalore - Export Promotion …14h ago
-
Senior-level Full TimeIndia - Bangalore - 5th floor, …14h ago