Associate Director, Generative AI Evaluation & Standards
ES022 Janssen Madrid, Spain
EUR 60K-80K (estimate) Mid-level Full Time
Tasks
- Attract develop and retain AI evaluation talent
- Build and lead GenAI evaluation team
- Build automated evaluation pipelines
- Collaborate on data quality rubric
- Communicate evaluation implications to stakeholders
- Conduct platform layer evaluation and security assessments
- Contribute to internal evaluation newsletter
- Create gold standard validation datasets
- Define benchmarks for data quality
- Define benchmarks for operational reliability
- Define benchmarks for regulatory compliance
- Define benchmarks for scientific validity
- Design evaluation frameworks for generative AI
- Enforce production ready standards
- Establish domain specific evaluation standards
- Lead evaluation governance board
- Maintain evaluation criteria for LLM quality
- Partner with QMS and MLOps for quality monitoring handoff
- Present evaluation findings to steering committee
- Publish evaluation standards and rubrics
- Set evaluation gates for development stages
- Train and document evaluation capability for teams
Perks/Benefits
- N/A
Skills/Tech-stack
Agent Reliability | Agentic AI | Automated Evaluation | Benchmarking | CDISC | Clinical data | Data Governance | Data Quality | Data Science | EMA | Evaluation Frameworks | FDA | FHIR | Generative AI | Gold Standard | Gold Standard Datasets | Knowledge graphs | Language Models | Large Language Models | Machine Learning | Multimodal AI | Ontology | Prompt engineering | Regulatory Compliance | Retrieval-Augmented Generation | Safety | Scientific accuracy
Education
Bachelor of Engineering | Bachelor of Science | Master of Engineering | Master of Science | PhD
Roles
Associate | Associate Director | Data Science | Data Science Manager | Director | Manager
Related jobs
-
Marketing Analytics Manager EUR 40K-45KCampaign Optimization | Data Visualization | Experimental Design | Incrementality analysis | Key Performance IndicatorsAccess to learning resources | Annual leave | Dental cover | Disability and life insurance | Employee stock purchase planMid-level Full TimeBarcelona, Spain8h ago
-
Head of Data Science EUR 70K-90KA/B | A/B Testing | AI | B testing | Causal InferenceEnglish classes | Event discount | Flexible compensation via Cobee | Flexible hybrid model | Gym membershipExecutive-level Full TimeMadrid R22h ago
-
Freelance Data Science Engineer (Python & SQL) USD 68K-68KDeep learning | Feature Engineering | Langchain | MLOps | Machine LearningEnglish proficiency requirement | Part-time project-based work | Portfolio based selectionMid-level FreelanceSpain - Remote R4d ago
-
Sr Manager Data Science and AI EUR 65K-78KAnalytics engineering | Artificial Intelligence | Business Requirements | Cross-functional | Cross-functional leadershipSenior-level Full TimeESP - Madrid, Spain4d ago
-
Tech AI/ML Product Manager - Job Hunt Engine EUR 75K-85KA/B | A/B Testing | Artificial Intelligence | B testing | CausalityEquity | Remote work | Sick leave | Vacation and local holidays | Work-life balanceSenior-level Full TimeSpain - Remote R5d ago
-
Data Science Manager EUR 62K-81KAWS | Azure | CI/CD | Cloud Computing | DockerDiscounts | Generous time off | Learning resources | Private health insurance | Referral bonusMid-level Full TimeBarcelona, ES; Madrid, ES; Mallorca, ES; …7d ago
-
Manager Data Strategy & Governance (Zaragoza) EUR 40K-43KAmazon Web Services | Artificial Intelligence | Cloud platform | Collibra | Data CatalogContinuous learning | Flexible schedule | Hybrid work | Multicultural environment | Travel occasionallyMid-level Full TimeZaragoza, Spain R8d ago
-
Engineering Manager - Data Platform EUR 30K-48KAgile | Big Data | Cloud Data | Cloud Data Platforms | Data EngineeringContinuous learning | Coworking space access | Health insurance | Home-office allowance | MentorshipMid-level Full TimeBarcelona8d ago
-
Data Engineering Manager EUR 60K-85KAirflow | Apache Flink | Apache Spark | BigQuery | CI/CDAnnual retreat | Generous PTO | Learning allowance | Remote work | Wellness allowanceSenior-level Full TimeSpain R8d ago
-
AI Engineering Manager EUR 50K-75KAnalytics | Artificial Intelligence | Client Relationship Management | Client relationship | Cloud ComputingFlexible work culture | International team | Yearly education budget | Yearly sport budgetMid-level Full TimeBarcelona, Catalonia, Spain8d ago
-
Senior Analytics Engineering Manager EUR 126KArchitecture Design | Big Data | Data Governance | Data Visualization | Data WarehouseFlexible working hours | Learning budget | MacBooks provided | On-call support | Work from home setupSenior-level Full TimeBarcelona R11d ago
-
Analytics & Insights Manager (Data Engineering) EUR 34K-34KAirflow | CI/CD | DBT | Data Engineering | Data ObservabilitySenior-level Full TimeMadrid Osiris, Spain12d ago
-
Data Scientist Manager EUR 46K-63KA/B | A/B Testing | AWS | Algorithmic Explainability | AzureCareer growth | Collaborative team | Innovation driven work environmentMid-level Full TimeCIUDAD BBVA [00018800 BCO. BILBAO VIZCAYA …13d ago
-
Amazon Neptune | Artificial Intelligence | Biomedical Ontologies | Clinical data | Cloud ComputingDisability insurance | Kindergarten card | Language classes | Life insurance | Paid vacationMid-level Full TimeR06-Barcelona, Spain13d ago
-
Product Manager AI Solutions EUR 65K-75KAI Foundry | Artificial Intelligence | Copilot Studio | Cross-Functional Collaboration | Cross-functionalMid-level Full TimeBarcelona, ES, 0890814d ago
-
Data Quality Manager Consultant EUR 50K-70KBigQuery | Business Intelligence | Cloud platform | Data Governance | Data LineageCertifications | Flexible remuneration | Health insurance | Hybrid work | Remote work optionMid-level Full TimeMadrid, Spain R20d ago
-
AI Engineering & Platform Lead EUR 72K-95KAzure Databricks | Azure Machine Learning | CI/CD | Computer Vision | Cost attributionCollaborative international team | Flexible working culture | Yearly education budget | Yearly sport budgetSenior-level Full TimeBarcelona, Catalonia, Spain21d ago
-
Product Manager, Data Analytics EUR 52K-68KAnalytics | Business Intelligence | Business Strategy | Customer Research | Data AnalysisAutonomy and flexibility | Bonding Leave | Charitable days | Coworking space | Learning platforms accessMid-level Full TimeSpain29d ago
-
Automation | Capacity Planning | Dashboard Development | Experimentation | Machine LearningEquipment provided | Fully remote | Parental leave | Training and development budget | Unlimited vacationMid-level Full TimeSpain - Remote R1mo ago
-
AI strategy | Artificial Intelligence | LLM orchestration | Language Models | Large Language ModelsAccess to training | Company equity (RSUs) | Comprehensive benefits | Counseling | Dedicated learning programMid-level Full TimeMadrid, Spain1mo ago
-
Azure | Data Quality | Data pipeline | Databricks | DynatraceSenior-level Full TimeBarcelona (Head Office), Spain1mo ago
-
Azure | Dashboarding | Data Governance | Data Quality | Data pipelineSenior-level Full TimeBarcelona (Head Office), Spain1mo ago
-
AI Program Manager EUR 54K-66KAI | Cloud Platforms | Product Management | Program Management | Project PlanningCareer development opportunities | Flexible working | Health benefits | Hybrid work model | Parental leaveMid-level Full TimeBARCELONA, B, ES, 080051mo ago
-
Data Science Manager EUR 39K-50KData Visualization | Feature Engineering | Machine Learning | Python | SQLMid-level Full TimeMadrid, SPAIN, Spain1mo ago
-
Data Science Manager EUR 37K-45KData Analysis | Data Visualization | Machine Learning | Python | SQLCareer growth | Hybrid workMid-level Full TimeMadrid, SPAIN, Spain1mo ago