Head of AI Evaluation & Reliability Engineering
Maharashtra, Pune, India
INR 1650K-6000K (estimate) Executive-level Full Time
Tasks
- Build AI evaluation reliability engineering capability
- Create benchmark packs golden datasets regression suites
- Define AI evaluation testing release gating runtime monitoring standards
- Design reusable AI evaluation frameworks
- Design runtime drift degradation monitoring systems
- Establish benchmark coverage versioning standards
- Implement edge case capture and benchmark expansion processes
- Lead AI reliability observability and evaluation teams
- Partner with platform product and solution engineering teams
- Perform failure mode analysis incident diagnostics
- Run continuous evaluation improvement loops
- Serve as internal SME for AI reliability benchmark design and evaluation methodology
- Set human review escalation pathways
Perks/Benefits
- N/A
Skills/Tech-stack
Artificial Intelligence | Benchmarking | CI/CD | Golden datasets | Governance | Human-in-the-loop | Incident Management | Language Models | Large Language Models | Machine Learning | Multimodal Models | Observability | Policy testing | Quality Assurance | RAG | Regression testing | Release gating | Retrieval-Augmented Generation | Runtime Monitoring | Safety testing | Telemetry | Test automation | The Loop | Tracing
Education
N/A
Related jobs
-
Senior Engineering Manager, Cloud Storage INR 2000K-4500KArtificial Intelligence | Benchmarking | C plus plus | C# | Cloud platformSenior-level Full TimeHyderabad, Telangana, India3h ago
-
AWS | Agile | Azure DevOps | CI/CD | Change ManagementSenior-level Full TimePune - Hinjewadi (Ascendas), India14h ago
-
Manager, Decision Sciences INR 520K-520KDeep learning | Fraud Detection | GANs | Generative AI | Gradient boostingAccident insurance | Annual health checkup | Annual leave | Business travel insurance | Cab facilityMid-level Full TimePune - Business Bay, India14h ago
-
Engineering Manager, Data Platform INR 1500K-2100KApache Airflow | Apache Spark | CI/CD | Call Management | Code QualityMid-level Full TimeIndia Remote R14h ago
-
Head of AI & Data Platform DevSecOps INR 1500K-5199KAPI Development | Alerting | Angular | Automation | Cause analysisFlexible benefits package | Travel up to 10 percentExecutive-level Full TimeGH Office: Chennai, India (Keppel)14h ago
-
DI Analyst INR 2520K-3500KData Modeling | Data Quality | Data Visualization | Data Warehousing | ETLReward and recognition | Volunteering opportunities | Wellness | Work-life balanceSenior-level Full TimeMumbai, India1d ago
-
Senior-level Full TimeGurugram, Haryana, India1d ago
-
Analytics | Artificial Intelligence | CRM | Cloud Data | Cloud Data PlatformsEntry-level Full TimeBengaluru, KA, India1d ago
-
Head of Data Science - Gurugram INR 1000K-2000KCausal Inference | Data Modeling | Data Science | Demand forecasting | ExperimentationWork from OfficeMid-level Full TimeGurgaon (Gurugram), Haryāna, India1d ago
-
AM CPAM - Digital and Data Analytics INR 3000K-4000KAnomaly Detection | Azure | Change Management | DAX | Dashboard DesignSenior-level Full TimeMumbai, Maharashtra, India1d ago
-
Senior Principal Product Manager - Supply Chain Analytics INR 2500K-3000KAI machine learning | Analytics Product Management | Backlog Management | Change Management | Data GovernanceSenior-level Full TimeHyderabad, India1d ago
-
Data Quality Assurance Manager INR 500K-1050KAnalytics | Bash | Data Pipelines | Data Quality | Data ValidationWork-life balanceMid-level Full TimeKarnataka, Bengaluru, India1d ago
-
AI Automation | Acceptance criteria | Agile | Analytics | Artificial IntelligenceMid-level Full TimeNoida, India1d ago
-
Strategic Analytics - Vice President INR 3300K-4000KAI machine learning | Apache Spark | Data Governance | Data Product Management | Data QualityExecutive-level Full TimeBengaluru, Karnataka, India1d ago
-
Senior Manager, Machine Learning INR 3000K-4600KAWS | Artificial Intelligence | Azure | Cloud Architecture | Cloud platformGenerous time off | Healthcare | Parental leave | Remote work | Retirement savings programSenior-level Full TimeRemote - India R1d ago
-
Senior Manager, Commercial, GBS Process & Analytics Services INR 2486K-3250KAgile | BPM | Decision Intelligence | Intelligent Automation | Machine LearningFlexible work arrangementSenior-level Full TimeIndia - Bangalore1d ago
-
EY - GDS Consulting - AIA - Gen AI - Manager INR 1600K-2500KAI Governance | Amazon Web Services | Artificial Intelligence | Azure | Cloud platformCareer development | Coaching | Feedback | Flexible work arrangement | Global initiativesMid-level Full TimeBengaluru, KA, IN, 5600161d ago
-
Data Engineering Manager INR 700K-1000KApache Airflow | Apache Flink | Apache Hadoop | Apache Hive | Apache IcebergMid-level Full Time110380-IND-BENGALURU-INTL BLR Twr-1&2 CARNATION, India1d ago
-
Director Of Engineering - Java, React, RAG & GenAI INR 1500K-6000KAI Observability | Agile | Azure OpenAI | CI/CD | Cloud NativeExecutive-level Full TimeHyderabad, India1d ago
-
IN_Manager_Data Engineer_D&A_Advisory_Bangalore INR 1500K-2000KAzure | CI/CD | Databricks | Delta Lake | DockerFlexibility programmes | Inclusive benefits | Mentorship | Wellbeing supportMid-level Full TimeBengaluru Millenia, India1d ago
-
IN_Manager_Data Engineer_D&A_Advisory_Bangalore INR 1500K-2000KAzure | CI/CD | Databricks | Delta Lake | DockerFlexibility programmes | Inclusive benefits | MentorshipMid-level Full TimeBengaluru Millenia, India1d ago
-
IN_Manager_Data Engineer_D&A_Advisory_Bangalore INR 1500K-2000KAzure | CI/CD | Databricks | Delta Lake | DockerFlexible work arrangements | Inclusive benefits | MentorshipMid-level Full TimeBengaluru Millenia, India1d ago
-
IN_Manager_Data Engineer_D&A_Advisory_Bangalore INR 1500K-2000KAzure | CI/CD | Databricks | Delta Lake | DockerFlexibility programs | Inclusive benefits | Mentorship | Wellbeing supportMid-level Full TimeBengaluru Millenia, India1d ago
-
IN_Manager_Data Engineer_D&A_Advisory_Bangalore INR 1500K-2000KAzure | CI/CD | Databricks | Delta Lake | DockerFlexibility programmes | Inclusive benefits | MentorshipMid-level Full TimeBengaluru Millenia, India1d ago
-
Senior Manager, Commercial, GBS Process & Analytics Services INR 2486K-3250KAI | Agentic AI | Agile | BPM | Change ManagementSenior-level Full TimeIndia - Bangalore1d ago