Associate Director, Generative AI Evaluation & Standards
ES022 Janssen Madrid, Spain
EUR 52K-80K (estimate) Mid-level Full Time
Tasks
- Assess agent reliability
- Assess data quality for evaluation rubric
- Attract, develop, and retain talent
- Build automated evaluation pipelines
- Build internal evaluation capability
- Collaborate with MLOps on handoff
- Conduct security assessments
- Coordinate domain evaluation standards
- Define LLM quality benchmarks
- Define RAG performance benchmarks
- Define validation datasets
- Design evaluation frameworks
- Develop therapeutic area evaluation criteria
- Drive scientific rigor culture
- Enforce production ready standards
- Establish evaluation governance
- Evaluate safety
- Lead GenAI evaluation and standards team
- Maintain evaluation frameworks
- Perform platform layer evaluation support
- Present evaluation findings
- Publish evaluation standards and rubrics
- Set evaluation gates
- Verify scientific accuracy
Perks/Benefits
Skills/Tech-stack
Agentic AI | Benchmarking | CDISC | Data Governance | Data Privacy | Data Quality | EMA | Evaluation Framework | Evaluation Pipeline | FDA | FHIR | Generative AI | Knowledge graphs | Language Models | Large Language Models | Multi-Modal | Multi-modal AI | Ontology | Prompt engineering | RAG | Regulatory Compliance | Retrieval-Augmented Generation | Safety evaluation | Scientific accuracy
Education
Roles
AI | AI Engineer | Associate | Associate Director | Data Scientist | Director | Engineer | Scientist
Related jobs
-
Product Manager, Data Science EUR 52K-82KAgile | Agile tools | Backlog Management | Clinical Trial Operations | Clinical trialMid-level Full TimeES022 Janssen Madrid, Spain1d ago
-
AI Engineer - Operations EUR 36K-50KAI dev tools | Data Science | Dev Tools | Language Models | Large Language ModelsExperimentation with AI tools | Modern office in Barcelona | Urban sports club membershipMid-level Full TimeBarcelona, Spain1d ago
-
Senior Analytics Engineer EUR 54K-72KAutomation | CI/CD | Dashboards | Data Lineage | Data QualityFlexible work arrangements | Health insurance | Professional development supportSenior-level Full TimeSpain1d ago
-
Agile methodologies | Data Analysis | Finetuning | Generative AI | Language ModelsContinuous training | Flexible work arrangement | Social benefitsMid-level Full TimeMadrid, Spain1d ago
-
Data Engineer II EUR 51K-70KAirflow | Apache Beam | DBT | Data Governance | Data StreamingDigital disconnect days | Employee stock purchase plan | Global induction program | Talent and development programs | Volunteer daysEntry-level Full TimeValència, VC, Spain1d ago
-
APIs REST | Big Data | Data Pipelines | Generative Models | Language ModelsHealth insurance | Immediate start | Office snacks | Team environment | Workplace inclusivityMid-level Full TimeMadrid, Spain1d ago
-
Middle Prompt Engineer (Cupid Media) EUR 37K-43KAI | Dataset labeling | Error Analysis | Git | JavaScriptBonuses | Employee recognition | Health benefits | Remote work | Training supportMid-level Full TimeValencia, ES - Remote R1d ago
-
Prompt Engineer (Сupid Media) EUR 29K-38KAI moderation | Content Optimization | Language Model | Large Language Model | Prompt engineeringBonuses | Employee recognition | Health benefits | Remote work | Training reimbursementEntry-level Full TimeValencia, ES - Remote R1d ago
-
Data Scientist EUR 36K-45KData Pipelines | Embeddings | Feature Engineering | Generative AI | Language ProcessingCareer development | Continuous training | Flexible working | Innovative projectsSenior-level Full TimeMadrid, ES2d ago
-
Big Data | Business Analytics | Data Analysis | Data Processing | Feature EngineeringFlexible schedule | Part-time treatment | Professional development opportunities | Remote workMid-level FreelanceSpain - Remote R2d ago
-
Senior AI Engineer EUR 52K-72KAWS | Algorithms | CI/CD | Data Structures | DatabricksCollaborative culture | Community support | Flexible work environment | Learning opportunitiesSenior-level Full TimeBarcelona, ESP-CT, Spain2d ago
-
AI Model Evaluation | AI model | Computational Linguistics | Data Analysis | Language ModelsFlexible hours | Remote work | Training opportunitiesSenior-level Freelance Part TimeSpain R2d ago
-
Senior Generative AI Engineer EUR 60K-72KAI Agents | Azure | Cloud Computing | Data Engineering | GPU/CUDASenior-level Full TimeBarcelona, Spain3d ago
-
AI Engineer – Tenerife - ORBIDI EUR 36K-45KAI Model Integration | AI model | Anthropic | Automation | Data PipelinesCareer growth opportunities | Competitive benefits | Dynamic environment | Hybrid work | Participate in decision-makingEntry-level Full TimeSanta Cruz de Tenerife, Spain5d ago
-
AI Automation | Access Control | CI/CD | Cloud Platforms | ContainerizationDlocal houses | Flexible schedule | Language classes | Learning and development | Remote work optionsSenior-level Full TimeMadrid5d ago
-
Senior Data Engineer EUR 85K-115KAlerting | Apache Spark | Columnar data | Columnar data format | Data CatalogingExtra recharge days | Flexible work from home | Learning and development budget | Medical/Dental/Vision insurance | Relocation supportSenior-level Full TimeTallinn, Spain (Remote) R5d ago
-
Data Analytics Engineer EUR 38K-39KAirflow | Cloud Data | Cloud Data Warehouses | Dagster | Data ModelingFlexible work | Health support | Inclusive community | Online learning | Work-life balanceMid-level Full TimeBarcelona, CT, Spain5d ago
-
Airflow | Artificial Intelligence | Azure Data | Azure Data Factory | Cloud ComputingFlexible work hours | Professional development | Social benefitsSenior-level Full TimeBarcelona, Spain5d ago
-
Data Analytics Engineer EUR 38K-39KAirflow | Cloud Data | Cloud Data Warehouses | DBT | DagsterFlexible work | Health and wellness benefits | Inclusive community | Online learning | Referral schemeMid-level Full TimeMadrid, Spain5d ago
-
Senior Machine Learning & AI Infrastructure Engineer EUR 72K-105KCI/CD | Cloud Platforms | Containerization | Data Drift | Data QualityFlexible work schedule | Language classes | Learning and development | Remote work options | Social budgetSenior-level Full TimeMadrid5d ago
-
AI architectures | Agentic AI | Cloud AWS | Data Science | Generative AIFlexible work arrangement | Inclusive environmentMid-level Full TimeSpain - Barcelona6d ago
-
Senior Data Engineer - 3-6 Month Contract EUR 60K-72KAWS | Airflow | Apache Iceberg | BigQuery | DagsterCompany events | Employee assistance program | Family support program | Hybrid work | Language classesSenior-level Full TimeBarcelona, ES6d ago
-
Senior Data Engineer EUR 68K-85KAWS | Airflow | Big Data | Big Data Technologies | DagsterCompany events | Critical illness | Employee assistance | Family support | Hybrid workSenior-level Full TimeBarcelona, ES6d ago
-
Data Engineer II EUR 51K-70KAirflow | Apache Beam | BigQuery | Cloud Data | Cloud Data WarehouseDigital disconnect days | Employee stock purchase plan | Global induction program | Talent development programs | Volunteer daysEntry-level Full TimeValència, VC, Spain6d ago
-
Senior Data Engineer I EUR 60K-72KAirflow | Apache Beam | DBT | Data Governance | DockerDisConnect days | Flexible work arrangements | Global induction program | Stock purchase plan | Talent development programsSenior-level Full TimeValència, VC, Spain6d ago