AI Quality Engineer
Bengaluru, KA, India
INR 1800K-3000K (estimate) Senior-level Full Time
Tasks
- Build and maintain evaluation datasets with ground truth reference outputs and edge cases
- Build automated evaluation pipelines in CI CD
- Build evaluation frameworks for groundedness factuality relevance and task completion
- Build monitoring and alerting for production agent quality accuracy latency and error rates
- Conduct systematic prompt evaluation when prompts or models change
- Create test harnesses for LLM systems with mocks and fixtures
- Define quality metrics and thresholds for AI systems
- Define testing strategies for AI agents and agentic workflows
- Design behavioral test suites for non deterministic outputs
- Develop regression suites to detect quality degradation
- Diagnose quality failures and build regression tests
- Identify AI specific failure modes like hallucinations prompt injection drift and context limits
- Maintain test infrastructure and reporting dashboards
- Partner with engineers to define testability requirements during design
- Report quality metrics to stakeholders
- Review solution designs for quality and testability
- Track quality trends and trigger re evaluation when needed
- Translate acceptance criteria into test scenarios
Perks/Benefits
- N/A
Skills/Tech-stack
Application Insights | Azure | Behavioral Testing | CI/CD | Dataverse | Docker | Evaluation datasets | Git | GitHub | LLM Evaluation | Load Testing | Monitoring and Alerting | Prompt testing | Pytest | Python | RAG evaluation | REST APIs | Regression testing | SQL | TypeScript
Education
Bachelor of Engineering | Bachelor of Science | Master of Science | PhD
Roles
AI | AI Quality Engineer | Engineer | Quality Engineer | Software Engineer
Related jobs
-
Senior AI Engineer INR 3000K-5000KAPI deployment | Adversarial Testing | Auto Retraining | Azure ML | CI/CDSenior-level Full TimeBangalore6h ago
-
Azure Data | Azure Data Factory | CI/CD | Data Factory | Data ReconciliationSenior-level Full TimeBangalore, India6h ago
-
Senior Talend Developer / Lead Data Engineer INR 1500K-2700KAzure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure DevOpsSenior-level Full Timebengaluru, India7h ago
-
Solutions Architect INR 2400K-3500KADLS | AWS | Airflow | Azure | CassandraBootcamp training | Broadband reimbursement | Continuous learning opportunities | Medical insurance | Paid certificationsSenior-level Full TimeIndia - Remote R8h ago
-
Sr. SW Engineer, Machine Learning INR 2475K-3465KA/B | A/B Testing | Agent systems | B testing | CI/CDFinancial wellness support | Hybrid work | Mental health support | Time offSenior-level Full TimeBengaluru, India8h ago
-
Sr GIS Data Engineer -Python l AI ML l SQL INR 1800K-2500KArcGIS | Batch Processing | Data Pipelines | Data Quality | FMESenior-level Full TimeMumbai, MH, India8h ago
-
Mid-level Full TimeIT Centre Bengaluru GDTC, India12h ago
-
Senior Data Engineer INR 1500K-2500KAccess Control | Azure | Azure Data | Azure Data Factory | Azure SQLSenior-level Full TimeHyderabad, IN, 500 00812h ago
-
Java Technical Lead- Career INR 2000K-3000KAWS | Airflow | Apache Beam | Atlassian | AzureHealthcare package | Hybrid work | Learning platform with guided career tracks | Paid time offSenior-level Full TimeIND-Trivandrum-Equifax Analytics-PEC, India12h ago
-
IN_Senior Associate_Data Analytics / Data Engineering_Digital Integration_Advisory_Gurgaon INR 2000K-2600K.NET | Agile | Azure | Azure Databricks | Azure SQLSenior-level Full TimeGurugram 10 C, India12h ago
-
IN_Senior Associate_Data Analytics / Data Engineering_Digital Integration_Advisory_Gurgaon INR 2000K-2600K.NET | Agile | Azure | Azure Databricks | C#Senior-level Full TimeGurugram 10 C, India12h ago
-
IN_Senior Associate_Data Analytics / Data Engineering_Digital Integration_Advisory_Gurgaon INR 2000K-2600K.NET | Agile | Azure | Azure Databricks | C#Flexibility programmes | Inclusive benefits | Mentorship | Wellbeing supportSenior-level Full TimeGurugram 10 C, India12h ago
-
IN_Senior Associate_Data Analytics / Data Engineering_Digital Integration_Advisory_Gurgaon INR 2000K-2600K.NET | Agile | Azure | Azure Databricks | C#Flexibility programs | Inclusive benefits | MentorshipSenior-level Full TimeGurugram 10 C, India12h ago
-
A/B | A/B Testing | AutoGPT | B testing | Dimensionality ReductionExecutive-level Full TimeMumbai Goregaon, India16h ago
-
Machine Learning Engineer IV INR 4443K-7998KAPIs | Amazon Web Services | Automated testing | Cloud Native | Cloud platformBonuses | Disability insurance | Employee resource groups | Inclusive culture | Life insuranceSenior-level Full TimeRemote, India R18h ago
-
Specialist, AI Solutions INR 900K-1300KData Analysis | Data Pipelines | Data Preparation | Documentation | ExperimentationAnnual health checkups | Catered lunch | Cellphone reimbursement | Corporate wellness programs | DaycareMid-level Full TimePune, Maharashtra, India21h ago
-
Mid-level Full TimeGurugram21h ago
-
Senior-level Full TimePune City, India22h ago
-
Senior AI Engineer INR 3000K-4500KAPI Development | Anthropic Claude | Bias Mitigation | BigQuery | Cloud ComputingContinued development opportunities | Discretionary bonus scheme | Enhanced maternity leave | Enhanced paternity leave | Flexible workingSenior-level Full TimeKarnataka22h ago
-
Mid-level Full TimePune, Maharashtra, India22h ago
-
AM - Cloud Data Engineer INR 1500K-2500KAWS N/A | Apache Airflow | Apache Kafka | Apache Spark | Azure DataClient-facing experience | Consulting experience | Growth mindset cultureSenior-level Full TimeBangalore, Karnataka, India22h ago
-
Senior-level Full TimeIN-TN-Chennai23h ago
-
AI Engineer – Reliability & Warranty Analytics INR 1800K-3000KBacktesting | BigQuery | CI/CD | Calibration | Cloud RunMid-level Full TimeChennai, Tamil Nadu, India23h ago
-
Associate III Business Intelligence-SUPPORT SERVICES-Data & Analytics - In House Engineering INR 800K-1800KBig Data | Big data concepts | C++ | Data Visualization | Data conceptsMid-level Full TimeBangalore, Karnataka, India1d ago
-
AI Engineer – Retail/CPG Domain INR 1800K-2500KAWS | Azure | Data Governance | Data Modeling | Data SecurityMid-level Full TimeGurugram, Haryana, India1d ago