Head of AI Evaluation & Reliability Engineering
Maharashtra, Pune, India
INR 1650K-6000K (estimate) Executive-level Full Time
Tasks
- Build AI evaluation reliability engineering capability
- Create benchmark packs golden datasets regression suites
- Define AI evaluation testing release gating runtime monitoring standards
- Design reusable AI evaluation frameworks
- Design runtime drift degradation monitoring systems
- Establish benchmark coverage versioning standards
- Implement edge case capture and benchmark expansion processes
- Lead AI reliability observability and evaluation teams
- Partner with platform product and solution engineering teams
- Perform failure mode analysis incident diagnostics
- Run continuous evaluation improvement loops
- Serve as internal SME for AI reliability benchmark design and evaluation methodology
- Set human review escalation pathways
Perks/Benefits
- N/A
Skills/Tech-stack
Artificial Intelligence | Benchmarking | CI/CD | Golden datasets | Governance | Human-in-the-loop | Incident Management | Language Models | Large Language Models | Machine Learning | Multimodal Models | Observability | Policy testing | Quality Assurance | RAG | Regression testing | Release gating | Retrieval-Augmented Generation | Runtime Monitoring | Safety testing | Telemetry | Test automation | The Loop | Tracing
Education
N/A
Related jobs
-
Techno Functional Data & Analytics Lead INR 1500K-2600KAI | AWS | Business Intelligence | Cloud platform | Customer SegmentationHybrid work model | Open to relocationSenior-level Full TimePerungudi, India9h ago
-
Director - AI Engineer INR 2400K-6000KAI Governance | AI Workflow Orchestration | AI workflow | API Development | Agent systemsExecutive-level Full TimeBangalore, Karnataka, India15h ago
-
Senior-level Full TimePune17h ago
-
Engineering Lead – AI & Developer Platform INR 3000K-5000KAPI Design | AWS | Anthropic | Authentication | CI/CDCertification sponsorship | Education sponsorship | Flexi-leaves | Health insurance | Work from home hybrid optionSenior-level Full TimeMumbai, India23h ago
-
Senior Analytics Manager - VP INR 2817K-5000KChange Management | Customer Segmentation | Data Quality | Data Science | Data WarehousingSenior-level Full TimeGurugram, DLF Downtown, India23h ago
-
Software Engineering Manager INR 1400K-1800KAgile | Behavior-Driven Development | Cloud Cost Management | Cost Management | DebuggingGuided career tracks | Healthcare package | Hybrid work setting | Online learning platform | Paid time offMid-level Full TimeIND-Pune-Equifax Analytics-PEC, India23h ago
-
Director Of Engineering - Java, React, RAG & GenAI INR 1500K-6000KAI Governance | AI Observability | Agile | Azure | Azure OpenAIExecutive-level Full TimeHyderabad, India23h ago
-
Manager - Data Scientist (AMVC) INR 1400K-2500KAnomaly Detection | CI/CD | Classification | Clustering | Data PreparationMid-level Full TimeMumbai Central Avenue, India23h ago
-
Senior Manager - QAC Audit Data Analytics Specialist -GCC INR 1081K-1968KData Integrity | Data Visualization | GxP | KPIs | Lean managementSenior-level Full TimeBengaluru Luxor North Tower, India23h ago
-
Apache Spark | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake StorageCollaborative work environment | Flexibility programmes | Inclusive benefits | Mentorship | Wellbeing supportSenior-level Full TimeBengaluru Millenia, India23h ago
-
Lead Data Scientist INR 3000K-4000KAWS | CI/CD | Crawling | Data Engineering | Data PreprocessingHealthcare | Paid volunteering days | Retirement planning | Wellbeing initiativesSenior-level Full TimeIND-Bangalore-TowerE,RMZ Infin, India23h ago
-
Data Science Manager INR 1500K-2700KAI Foundry | Airflow | Azure AI | Azure AI Foundry | Azure CognitiveAccident insurance | Career coaching | Diversity and inclusion | Employee-friendly policies | Health benefitsMid-level Full TimeGurugram, Haryana, India1d ago
-
Technology and Innovation - Data Analytics Manager INR 3000K-4000KAgile | Azure | Azure Data | Azure Data Lake | Azure Databricks401k with company match | Medical, dental, and vision insurance | PTOSenior-level Full TimeIndia1d ago
-
Manager, Platform Engineering - Databases INR 500K-1500KAutomation | CI/CD | Code review | Continuous Delivery | Continuous integrationMid-level Full TimeBangalore, Karnataka, IN India, 5600871d ago
-
Principal Software Engineering Manager - AI Engineering (Position located in Bengaluru, India) INR 3000K-3500KAWS Bedrock | AWS Glue | AWS Lambda | Anthropic Claude | BatchAdoption Assistance | Bonus | Certification completion bonus | Certification reimbursement | Employee referral bonusSenior-level Full TimeBengaluru, India3d ago
-
Airbyte | Airflow | BigQuery | Cloud Composer | Cloud StorageContinuous learning culture | Flexible PTO | Health, dental, and vision insurance | Inclusive and transparent work culture | Remote-friendly work environmentSenior-level Full TimeIndia3d ago
-
Digital & GenAI Analyst - Consulting INR 1000K-2500KAPI | Artificial Intelligence | Fine Tuning | Generative AI | LLM DeploymentMid-level Full TimeIndia3d ago
-
Agile | CI/CD | Cloud Computing | Cloud Native | DevOpsAsynchronous collaboration | Career development support | Continuous learning support | Flexible working hours | Fully remoteExecutive-level Full TimeIndia R3d ago
-
AI and Machine Learning Manager INR 1000K-1680KAgile | Data Analysis | Data Privacy | Data Security | Deep learningMid-level Full TimeBengaluru, Karnātaka, India3d ago
-
SAS DMS dbDocStore Engineering, VP INR 4000K-4500KAccess Control | Agile | Authentication | Capacity Planning | Cause analysisAccident and term life insurance | Best in class leave policy | Certification sponsorship | Childcare assistance reimbursement | Comprehensive hospitalization insuranceExecutive-level Full TimeBangalore, Velankani Tech Park, India3d ago
-
Director, Data Engineering INR 1500K-6000KARM Templates | AWS | AWS Glue | AWS Lambda | AWS RedshiftExecutive-level Full TimePune, India3d ago
-
AI and Machine Learning Manager INR 1000K-1680KAgile | Data Privacy | Data Security | Deep learning | Machine LearningCareer Development Programs | Health and wellbeing benefitsMid-level Full TimeBengaluru, Karnātaka, India3d ago
-
Manager - Data Engineer INR 2400K-3200KARM Templates | Agile | Azure Data | Azure Data Factory | Azure Data LakeDaycare support | Deferred compensation programs | Flexible work schedules | Leadership and training programs | Parental leave supportSenior-level Full TimeIN-KA-BENGALURU-NORTHGATE ~ Sy No 2/2 Venkatala …3d ago
-
Business Solutions Group Data Analytics, Vice President, Institutional Securities Management INR 1500K-5800KAlteryx | Artificial Intelligence | Automation | Dashboarding | Data EngineeringExecutive-level Full TimeCOMMERZ III, OBEROI GARDEN CITY, India3d ago
-
Data Science Engagement Lead INR 1800K-2400KAWS | Agile | Artificial Intelligence | Azure | CI/CDInclusive culture | Training and development programs | Workplace adjustmentsSenior-level Full TimeSHELL CENTRE – CHENNAI, India3d ago