Head of AI Evaluation & Reliability Engineering
Maharashtra, Pune, India
INR 1650K-6000K (estimate) Executive-level Full Time
Tasks
- Build AI evaluation reliability engineering capability
- Create benchmark packs golden datasets regression suites
- Define AI evaluation testing release gating runtime monitoring standards
- Design reusable AI evaluation frameworks
- Design runtime drift degradation monitoring systems
- Establish benchmark coverage versioning standards
- Implement edge case capture and benchmark expansion processes
- Lead AI reliability observability and evaluation teams
- Partner with platform product and solution engineering teams
- Perform failure mode analysis incident diagnostics
- Run continuous evaluation improvement loops
- Serve as internal SME for AI reliability benchmark design and evaluation methodology
- Set human review escalation pathways
Perks/Benefits
- N/A
Skills/Tech-stack
Artificial Intelligence | Benchmarking | CI/CD | Golden datasets | Governance | Human-in-the-loop | Incident Management | Language Models | Large Language Models | Machine Learning | Multimodal Models | Observability | Policy testing | Quality Assurance | RAG | Regression testing | Release gating | Retrieval-Augmented Generation | Runtime Monitoring | Safety testing | Telemetry | Test automation | The Loop | Tracing
Education
N/A
Related jobs
-
Analytics Solutions Manager INR 1234K-1748KAccess Control | Anomaly Detection | Artificial Intelligence | CDC | CI/CDMid-level Full TimeBengaluru, Karnataka, India7h ago
-
Engineering Manager – AI Platforms INR 26K-28KAWS Bedrock | AWS SageMaker | Agile | Azure AI | CI/CDCollaborative work environment | Global consulting experience | Leadership development | Training programsSenior-level Full TimeMumbai, MH, India9h ago
-
Senior Manager - Product Analytics (Individual Contributor) INR 2400K-2829KA/B | A/B Testing | B testing | Causal Inference | Cohort AnalysisSenior-level Full TimeBengaluru, India11h ago
-
Manager - Forward Deployed Engineer / AI Engineer INR 1800K-2500KAPI Integration | Agent Frameworks | Agile | Azure | BERTMid-level Full TimePune, Maharashtra, India11h ago
-
Manager - Forward Deployed Engineer / AI Engineer INR 1800K-2500KAgentic Frameworks | Agile | Azure | BERT | DjangoMid-level Full TimeBangalore, Karnataka, India11h ago
-
Director - Digital Data & AI INR 1800K-4500KAI Governance | Agentic Systems | Artificial Intelligence | Automation | Cloud ArchitectureExecutive-level Full TimeGurgaon, HR, IN11h ago
-
AI Engineering Lead INR 2500K-5000KAgent systems | Agentic behavior | Amazon Web Services | Azure | CI/CDCollaborative culture | Five-day work week | Flexible working hours | Supportive work environmentSenior-level Full TimeGurugram, Haryana, India15h ago
-
Senior-level Full TimePune, Maharashtra, India18h ago
-
Lead AI Engineer INR 2500K-4500KAWS | Agent Orchestration | Agentic AI | Artificial Intelligence | AzureSenior-level Full TimeMumbai, India18h ago
-
Senior Data Engineer - Manager INR 2400K-3500KAWS | AWS EMR | AWS Glue | AWS IAM | AWS IcebergEmployee networks | Flexible work/life support | Inclusive development opportunities | Paid volunteer daysSenior-level Full TimeBangalore, India18h ago
-
VP, Analytics – Marketing Measurement Consulting (L12) INR 500K-700KA/B | A/B Testing | AWS | B testing | BirstCareer advancement | Flexible work hours | Upskilling opportunities | Work from home optionExecutive-level Full TimeHyderabad IN, India18h ago
-
Mid-level Full TimeOff Embassy Golf Links Business Park, …18h ago
-
Alteryx | Artificial Intelligence | Controls compliance | Data Analysis | Data VisualizationSenior-level Full TimeMumbai, Maharashtra, India22h ago
-
Engineering Manager - GenAI studios INR 1200K-2000KCI/CD | Cassandra | Distributed Systems | DynamoDB | Google CloudMid-level Full TimeBengaluru1d ago
-
Senior Software Engineering Manager INR 2755K-3500KAWS | Agile | CI/CD | Clean architecture | Data EngineeringEnhanced medical benefits | Family wellbeing benefits | Paid time off | Wellbeing support | Work-life balanceSenior-level Full TimeHyderabad, India1d ago
-
Senior Manager – AI Architect INR 2000K-3800KAI Security | API Development | AWS | Adversarial Attacks | Azure AISenior-level Full TimeIndia-Gurugram1d ago
-
Engineering Manager – AI Platforms GBP 26K-28KAWS | AWS Bedrock | AWS SageMaker | Agentic Workflows | Agile ScrumCollaborative team environment | Leadership development programs | Training sessionsSenior-level Full TimeMumbai, MH, India1d ago
-
Director of Software Engineering - Big Data, Cloud INR 2000K-6000KAI machine learning | AWS | Agile | Azure | Big DataExecutive-level Full TimeMumbai, Maharashtra, India1d ago
-
Manager Accounting / Data Analytics Manager INR 800K-1200KArtificial Intelligence | Budgeting | Data Visualization | ERP | Financial ModelingMid-level Full TimeAurangabad, Maharashtra, India1d ago
-
Data Engineering Senior Manager - HIH - Evernorth INR 1500K-2000KAWS Cloud | AWS Fargate | AWS Glue | AWS IAM | AWS LambdaSenior-level Full TimeHIH - Hyderabad, India1d ago
-
Mgr, Advanced Analytics & Reporting 4 INR 2520K-3500KDashboarding | Data Governance | Data Modeling | Data Quality | Data WarehousingSenior-level Full TimeBangalore, India R1d ago
-
Sr Mgr Data Science Mktg INR 1960K-2600KAWS | Adobe Analytics | Artificial Intelligence | Automation | Business IntelligenceFlexible work environment | Internal mobility | Volunteering opportunities | Well-being | Work-life balanceSenior-level Full TimeKA Bangalore, India1d ago
-
Senior-level Full TimeINDIA TECHNOLOGY CENTER, India1d ago
-
Director, Data Engineering INR 1500K-6000KApplied Data Science | Architecture Review | Batch Processing | Cloud Data | Cloud Data ServicesExecutive-level Full TimePune, India1d ago
-
Senior Engineering Manager INR 3000K-4000KAWS | Agentic Systems | Agile | Amazon EMR | Amazon Elastic Container ServiceEnhanced medical benefits | Family well-being benefits | Health and wellbeing benefits | Paid time off | Reward and recognitionSenior-level Full TimeHyderabad, India2d ago