AI QA & Evaluation Engineer
Tasks
- Apply AI security and governance best practices
- Automate validation suites
- Create rubric based evaluation tasks
- Design test strategies for AI ML systems
- Document AI agent behavior and hallucinations
- Ensure AI compliance and data privacy
- Generate performance reports for AI models
- Implement AI ML regression testing
- Implement CI/CD pipelines for ML models
- Refine evaluation rubrics using feedback
- Run integration testing for ML models
- Validate AI data quality for training and inference
- Validate output grounding in verified data
Perks/Benefits
Skills/Tech-stack
AWS | Artificial Intelligence | Azure | Benchmarking | CI/CD | Cloud platform | GitHub Actions | Google Cloud | Google Cloud Platform | Integration Testing | LLM Evaluation | Langsmith | Language Models | Large Language Models | Machine Learning | Prompt engineering | Python | RAG | Regression testing | Retrieval-Augmented Generation | Terraform | Test automation | TypeScript
Education
N/A
Roles
AI | AI Evaluation Engineer | Engineer | Evaluation Engineer | QA Engineer
Related jobs
-
AI Engineer (Manager level) INR 1200K-2500KApache Spark | Azure AI | Azure CI | Azure CI/CD | Azure CognitiveMid-level Full TimeHyderabad, India1d ago
-
AI Engineering Manager INR 2000K-4000KAPI Integration | Automated testing | Claude API | Continuous Deployment | Continuous integrationMid-level Full TimeIndia, Bengaluru (Hybrid) R1d ago
-
API | JSON | Java | JavaScript | PythonOpportunity to work on AI projects | Remote workMid-level ContractIndia; Pakistan; Nigeria; Kenya; Egypt; Ghana; … R1d ago
-
Senior-level Full TimeBangalore, India Office (BANGALORE) R1d ago
-
Associate Engineer-AI/ML INR 2165K-3000KAPI Integration | Amazon Web Services | Azure | C++ | CD pipelinesEmployee discounts | Training opportunities | Tuition reimbursement | Wellness benefitsMid-level Full TimeBangalore - Karnataka, India - Kalyani …1d ago
-
Junior BI Developer/Data Engineer - India Home Base INR 300K-300KBig Data | Data Modeling | Data Pipelines | ETL | JavaFlexible remote work | Professional development opportunitiesEntry-level Full TimeBangalore, India1d ago
-
AWS | Asynchronous programming | Azure | Claude Code | FastAPISenior-level ContractShaikpet, India R1d ago
-
Access Control | Anomaly Detection | Audit Logging | CI/CD | ClassificationRemote working modelSenior-level Full TimeIndia1d ago
-
AI safety & Security controls, LLM Guardrails/Associate Director, Software Engineering Specialist INR 1400K-2400KAPI Development | Asynchronous processing | Audit trails | Cloud Computing | Content SafetyCareer development | Flexible working | Inclusive cultureMid-level Full TimePune, Maharashtra, India R1d ago
-
Databricks Data Engineer - Contract-to-Hire INR 1500K-3000KAWS | Apache Spark | Azure | Data Reconciliation | Data ValidationMedical insurance | Paid time off | Professional development trainingSenior-level Full TimePune, MH, India1d ago
-
Senior Databricks Data Engineer INR 1500K-3500KAWS | CI/CD | Databricks | Delta Lake | LakehouseMedical insurance | Paid time off | Performance incentives | Professional development trainingSenior-level Full TimePune, MH, India1d ago
-
Lead AI Specialist (Finance Operations) INR 3264K-5449KAccess Control | Agent Orchestration | Audit Logging | BigQuery | Change ManagementCollaborative culture | Growth opportunities | Work-life balance programsSenior-level Full TimeMumbai, MH, India1d ago
-
Senior-level Full TimeBengaluru, Karnataka, India R1d ago
-
Applied AI ML Associate INR 1400K-2400KAutomation | Data Quality | Data analytics | Data pipeline | DatabaseMid-level Full TimeBengaluru, Karnataka, India1d ago
-
Applied AI ML Associate INR 1400K-2400KCause analysis | Credit Risk | Credit risk modeling | Data Analysis | Data DriftMid-level Full TimeBengaluru, Karnataka, India1d ago
-
AES - DE - Generative AI Solution Architect INR 2500K-4800KAI Agents | Azure | Conversational AI | Copilot Studio | Language ModelsSenior-level Full TimePune, Maharashtra, India1d ago
-
Lead Assistant Manager INR 1500K-2500KApache Airflow | BigQuery | Cloud Composer | Cloud Storage | Cloud platformSenior-level Full TimePune, Maharashtra, India1d ago
-
Cybersecurity Architect III - Offensive Security, Databricks, AI INR 2000K-3500KAWS | Agile methodologies | Artificial Intelligence | Auditability | Continuous DeliverySenior-level Full TimeHyderabad, Telangana, India1d ago
-
Mid-level Full TimeBangalore, Karnataka, India R1d ago
-
AI Architect / Sr. AI Engineer INR 2000K-5000KAnthropic Claude | CI/CD | Chroma | Django | Django RESTRemote workSenior-level Full TimePune R1d ago
-
Senior Software Engineer(AI) INR 2500K-4580KAPI Development | Access Control | Agent Builder | Alerts | Automated testingBirthday off | Flexible working hoursSenior-level Full TimeGurgaon1d ago
-
Data Engineer Consultant Senior INR 3000K-4000KAgile | Alerting | Auditability | Automated testing | AzureBike to work | Career mentorship | Educational assistance | Employee assistance program | Employee discountSenior-level Full TimeBengaluru, KA, India R1d ago
-
Data Engineer Consultant Principal INR 3000K-4000KAccess Control | Agile | Audit and compliance | Auditability | Automated testingBike to work | Career progression mentorship | Educational assistance | Employee assistance programme | Employee discount schemeSenior-level Full TimeBengaluru, KA, India R1d ago
-
Senior Software Engineer I-Agentic AI INR 2500K-4580KAsynchronous Workflows | C++ | Containerization | Distributed Systems | DockerConference reimbursement | Employee assistance program | Flexible time off | Hybrid work | LinkedIn Learning accessSenior-level Full TimeHyderabad R1d ago
-
API Testing | CI/CD | Postman | Pytest | RESTCompany-paid certifications | Language courses | Medical insurance | Mental health support | Udemy accessSenior-level Full TimeIndia1d ago