Staff Technical Product Manager – AI/LLM expertise + AI Evaluation Science
Hyderabad, India
R
INR 3000K-4995K (estimate) Senior-level Full Time
Tasks
- Define custom evaluation metrics rubrics datasets and strategies
- Define evaluation approaches for AI agents task completion tool usage and workflows
- Develop reusable evals for domain behaviors and business outcomes
- Embed evals into AI lifecycle define measure build evaluate improve release monitor
- Enable continuous evaluation in production
- Establish AI evaluation as core engineering capability
- Establish quality gates for AI applications
- Identify gaps in evaluation capabilities
- Influence platform roadmap based on enterprise needs
- Integrate evals into development regression and release workflows
- Partner with AI platform teams to implement evaluate and drive eval adoption
- Require teams to define expected behavior and measurable quality criteria
- Validate out of the box evals
Perks/Benefits
- N/A
Skills/Tech-stack
AI Observability | Agent evaluation | Artificial Intelligence | CI/CD | Continuous Evaluation | Developer platforms | Evaluation | Evaluation datasets | LLM-as-a-Judge | Language Models | Large Language Models | Metrics | Platform strategy | Product Management | Quality Engineering | Rubrics
Education
N/A
Related jobs
-
Testing Engineering Specialist Advisor INR 2475K-3380K.NET | AI Search | AWS | Application Insights | AzureSenior-level Full TimeNoida, UP, IN R22h ago
-
Mid-level Full TimePune R1d ago
-
Insight & Analytics Manager GBP 31K-31KArtificial Intelligence | Automation | Data Governance | Data Quality | Data ScienceCycle to work scheme | Hybrid working | Life assurance | Paid volunteering day | PensionMid-level Full TimeDarwen - India Mill Centre, United … R1d ago
-
Associate Manager, Competitive Intelligence, Portfolio Analytics INR 1377K-2541KArtificial Intelligence | Benchmarking | Business Development | Clinical Development | CommercializationMid-level Full TimeHyderabad, India R1d ago
-
Mid-level Full TimeBangalore, Karnataka, India R2d ago
-
AI Agents | Automated testing | CI/CD | Cloud Computing | Distributed SystemsCareer advancement | Collaborative global teams | Flexible working arrangements | Fully remote | Professional developmentSenior-level Full TimeIndia R2d ago
-
Senior Engineering Manager - Data Services INR 2500K-4300KAWS | Agile | Apache Hadoop | Apache Iceberg | Apache KafkaCareer development access | Employee resource groups | Flexible work from home | Generous PTO policy | Paid volunteer timeSenior-level Full TimeIndia-Bangalore R2d ago
-
Manager, PV Analytics Center of Excellence INR 1524K-2541KAI | Business Objects | Cognos | Data Integrity | DatabricksMid-level Full TimeHyderabad, India R2d ago
-
Artificial Intelligence | CRM management | ChatGPT | Claude | DashboardingHybrid work schedule | Inclusive work culture | Professional development opportunities | Supportive leadersSenior-level Full TimeTower A (GOSS) DLF Building No. … R2d ago
-
Principal Advisor, GenAI Technical Product Management INR 3000K-5000KAI Governance | APIs | Agentic AI | Cloud Architecture | Context engineeringHybrid work | Work-life flexibilitySenior-level Full TimeBengaluru, India R2d ago
-
Artificial Intelligence | Customer Relationship Management | Customer relationship | Dashboard Development | Data IntegrityHybrid work | Inclusive culture | Professional development opportunitiesSenior-level Full TimeGurugram - DLF Building, India R2d ago
-
AI Engineering Manager INR 2400K-4408KAPIs | Agent tooling | Automated testing | CI/CD | Claude APIMid-level Full TimeIndia, Bengaluru (Hybrid) R6d ago
-
Data Engineering Lead INR 2000K-4580KAWS | Apache Airflow | Apache Spark | CI/CD | DBTBike to work | Career progression | Educational assistance | Employee assistance programme | Employee discountSenior-level Full TimeMumbai, MH, India R6d ago
-
FireFly - AI Engagement Manager INR 1000K-2000KAWS | Adobe AJO | Adobe Creative | Adobe Creative Cloud | Adobe Creative Cloud APIsCareer development | Hybrid work flexibilityMid-level Full TimeNoida, India R7d ago
-
Software Engineer - Front Line Manager INR 1800K-2525KAirflow | Angular | Apache Beam | BigQuery | BigtableGuided career tracks | Healthcare packages | Hybrid work setting | Online learning platform | Organizational growth potentialMid-level Full TimeIND-Trivandrum-Equifax Analytics-PEC, India R7d ago
-
Manager of Software Engineering - Java Full stack developer INR 2000K-3200KAPI first | AWS | CI/CD | Databricks | DockerSenior-level Full TimeHyderabad, Telangana, India R9d ago
-
Data & AI Delivery Leader INR 3242K-4680KAnalytics | Artificial Intelligence | BigQuery | CI/CD | Cloud DataCertification reimbursement | Employee resource groups | Helping hands fund | Inclusive wellbeing program | Learning and development supportSenior-level Full TimeBangalore IBC Knowledge Park, India R12d ago
-
Sr. Manager - Data Engineer INR 2771K-4064KAWS CloudWatch | AWS ECS | AWS EKS | AWS EMR | AWS GlueHybrid work arrangement | Work-life flexibilitySenior-level Full TimeHyderabad, India R12d ago
-
Portfolio Manager - Data, AI & Analytics INR 1000K-2000KAdvanced Analytics | Artificial Intelligence | Budget Forecasting | Business Intelligence | Cloud DataHybrid work options | Remote work flexibility | Travel for work as requiredMid-level Full TimeHyderabad, TG, IN R14d ago
-
AI Product Development | Agile | Amazon Web Services | Azure | Cloud ComputingEducational assistance | Life insurance | Medical insurance | Personal accident insurance | Training coursesMid-level Full TimeIND-HYNB-Hyderabad, India R14d ago
-
Principal Analyst - AI Portfolio / AI Program INR 2000K-5000KAI Governance | Advanced Analytics | Agentic Workflows | Artificial Intelligence | Business ProcessSenior-level Full TimeIND - KA - Bangalore - … R14d ago
-
Manager, Software Development Engineering (Cloud Security / SASE / Data Analysis / Threat Intelligence / LLM) INR 3600K-4600KAPI | AWS | Azure | Cloud Architecture | Cloud platformEducation reimbursement | Health plans | Hybrid work model | In-office perks | Parental leaveSenior-level Full TimeMohali, IND R16d ago
-
Software Development Mgmt 5 INR 1500K-6000KAWS | AWS Lambda | Amazon EMR | Apache Hudi | Apache IcebergExecutive-level Full TimeIndia - Bangalore - Remote Office R16d ago
-
Senior-level Full TimeIND19-01-Bengaluru-EPIP 122 (Phase II), India R18d ago
-
Training Program Management INR 3000K-4000KAgentic AI | Agile | Artificial Intelligence | Cause analysis | Change ManagementExecutive-level Full TimeECN02 - Bengaluru, Kalayani Platina (ECN02), … R22d ago