Associate Director, Generative AI Evaluation & Standards
ES022 Janssen Madrid, Spain
EUR 52K-80K (estimate) Mid-level Full Time
Tasks
- Build automated evaluation pipelines
- Collaborate on data quality rubric
- Conduct platform layer evaluation
- Create evaluation training materials
- Define benchmarks
- Design evaluation frameworks
- Develop domain specific criteria
- Enforce production ready standards
- Ensure regulatory compliance
- Establish gold standard datasets
- Lead and manage evaluation team
- Lead evaluation governance
- Maintain evaluation standards
- Partner on production monitoring handoff
- Perform security assessments
- Present evaluation findings
- Publish evaluation rubrics
- Set evaluation gates
- Validate scientific accuracy
Perks/Benefits
Skills/Tech-stack
Agentic Systems | Benchmarking | CDISC | Cybersecurity | Data Governance | Data Privacy | Data Quality | Evaluation Framework | Evaluation framework design | FHIR | Framework design | Generative AI | Knowledge graphs | Language Models | Large Language Models | MLOps | Multi-Modal | Multi-modal AI | Ontologies | Prompt engineering | RAG | Regulatory Compliance | Retrieval-Augmented Generation | Scientific accuracy
Education
Related jobs
-
HQ - Senior Analytics Engineer EUR 49K-79KAmazon Redshift | BI Dashboards | DBT | Data Lineage | Data ModelingMeaningful benefits | Remote work optionSenior-level Full TimeMadrid HQ, ES R1d ago
-
AI Safety | API Design | Amazon Bedrock | Automated testing | Azure OpenAIBike leasing | Discount on products | Hybrid working | Option to work abroad | Subsidised transportSenior-level Full TimeMadrid, Spain1d ago
-
Senior AI Engineer - SDLC (All genders) EUR 60K-86KAmazon Bedrock | Azure OpenAI | CI/CD | Distributed Systems | DockerBike leasing | Employee discount | Hybrid working | Subsidised transport | Training and developmentSenior-level Full TimeMadrid, Spain1d ago
-
API Design | API Testing | Amazon Bedrock | Azure OpenAI | Context ManagementBike leasing | Employee product discount | Hybrid working | Option to work abroad | Subsidised transportSenior-level Full TimeMadrid, Spain1d ago
-
Senior Software AI Engineer - SDLC (All Genders) EUR 60K-88KCI/CD | Code generation | Context engineering | Docker | EmbeddingsBike leasing | Discount on products | Hybrid working | Option to work abroad | Subsidised transportSenior-level Full TimeMadrid, Spain1d ago
-
AI Engineering | Artificial Intelligence | Competency Mapping | Curriculum Development | Data ScienceCross-cultural networking | Free training | Opportunity for public speaking | Remote workSenior-level Part TimeMadrid, Spain, ES1d ago
-
Machine Learning Engineer EUR 67K-93KAWS | Airflow | Audio Processing | Azure | CI/CDHealthcare, Dental and Vision coverage | Ownership and autonomy | Rapid iteration | Work with top engineersMid-level Full TimeSpain1d ago
-
Data Analyst / Engineer EUR 49K-61KAzure Data | Azure Data Factory | Azure Synapse | Azure Synapse Analytics | DAXEntry-level Full TimeMadrid, MD, Spain1d ago
-
Benchmarking | Computer Vision | Diffusion Models | Foundational models | Generative AIBiannual company retreats | Comprehensive parental leave | Flexible remote work | Full tech equipment provided | Monthly in person collaboration sessionsExecutive-level Full TimeSpain1d ago
-
Senior Data Engineer EUR 60K-63KAPI Integration | Alerting | Anomaly Detection | Auditability | Azure DataFlexible working environment | Inclusive workplace | Work-life balanceSenior-level Full TimeMadrid, Europe, Spain2d ago
-
Entry-level Full TimeMadrid, MD, Spain2d ago
-
Anomaly Detection | Automated reporting | Dashboard Development | Data Modeling | Data PipelinesContinuous learning | Education support | Employee discounts | Family-friendly benefits | Flexible time offMid-level Full TimeSpain2d ago
-
AWS | Apache Flink | Apache Kafka | Apache Spark | Batch ProcessingEquity opportunities | Flexible time off | Fully remote | Global health coverage | Home office budgetSenior-level Full TimeSpain R2d ago
-
Data Scientist EUR 64K-81KAmazon SageMaker | Azure Machine Learning | Cross-validation | DVC | Data PipelinesCoaching | Hybrid work | Training and development opportunities | Wellness programsEntry-level Full TimeMadrid HQ (KES51610), Spain2d ago
-
Sr Manager – Data Science & AI - UK & AUS EUR 54K-72KArtificial Intelligence | Data Governance | Data Privacy | Data Science | Data VisualizationSenior-level Full TimeESP - Madrid, Spain2d ago
-
AWS | Artificial Intelligence | Azure | Blue Yonder | Cloud platformComprehensive insurance | Diversity and inclusion networks | Extra vacation days | Flexible schedules | Hybrid work modelMid-level Full TimeBarcelona, La Rotonda, Spain2d ago
-
Data and Process Engineer EUR 42K-84KAI Agents | API Development | Artificial Intelligence | Backend Development | Cloud ComputingBonus scheme | Health insurance | Hybrid working | Monday to Friday work pattern | Pension schemeEntry-level Full TimeMadrid, Spain2d ago
-
Quality Data Analytics Manager EUR 55K-72KAI analytics | Dashboards | Data Governance | Data Integrity | Data ManagementFlexible hybrid working | Growth opportunities | Individual development plans | Training opportunitiesMid-level Full TimeAlicante, Spain2d ago
-
Sr. Tech AI/ML Product Manager - Job Hunt Engine EUR 68K-90KA/B | A/B Testing | Artificial Intelligence | B testing | CausalityEquity | Remote work | Sick leave | Vacation and local holidays | Work-life balanceSenior-level Full TimeSpain - Remote R2d ago
-
Tech Lead en IA EUR 36K-45KAzure Foundry | Azure Foundry AI | Copilot | MLOps | PythonFlexible compensation | Flexible working hours | Inclusive workplace | Indefinite contract | Positive work environmentSenior-level Full TimeBarcelona, CT, Spain R3d ago
-
Senior Data Engineering (Tech Lead) EUR 54K-84KAI Agent | AWS Glue | AWS Lambda | AWS Step Functions | Amazon AthenaChildcare benefits | Company-paid volunteer days | Flexible hours | Gym subscription | Hybrid work modelSenior-level Full TimeMadrid, MD, Spain3d ago
-
Senior AI Engineer ( All Genders) EUR 62K-90KAPI Design | Amazon Bedrock | Automated testing | Azure OpenAI | Context ManagementAnnual vacation days | Bike leasing | Employee discount | Hybrid working | Private medical health insuranceSenior-level Full TimeMadrid, Spain3d ago
-
Senior AI Engineer ( All Genders) EUR 62K-90KAPI Design | Amazon Bedrock | Automated testing | Azure OpenAI | Context ManagementAnnual vacation days | Bike leasing | Hybrid working | Private medical health insurance | Product discountSenior-level Full TimeMadrid, Spain3d ago
-
Senior Data Engineer - AI Platform EUR 56K-85KAPI Design | Audit Logging | Authentication | Authorization | BigQueryHybrid work model | Open Source friendly | Training and certificationSenior-level Full TimeMadrid3d ago
-
Senior AI Engineer (All Genders) EUR 60K-88KCI/CD | Context engineering | Docker | Embeddings | JavaAnnual vacation days | Hybrid working | Private medical health insurance | Product discount | Subsidised transport or bike leasingSenior-level Full TimeMadrid, Spain3d ago