AI QA Trainer - LLM Evaluation
Tasks
- Analyze model behavior
- Assess hallucination detection
- Collaborate on evaluation metrics and quality monitoring
- Conduct red teaming
- Define pass fail criteria
- Design evaluation frameworks
- Develop evaluation rubrics
- Evaluate large language models
- Execute test plans
- Identify model failures
- Maintain documentation and issue reports
- Perform adversarial testing
- Recommend prompt and guardrail improvements
- Run regression suites
- Support test automation using Python and SQL
- Validate tool usage accuracy
- Verify grounding and retrieval augmented generation
Perks/Benefits
Skills/Tech-stack
AI Safety | Adversarial Testing | Bias Assessment | Experiment tracking | Factual consistency | Hallucination detection | Information Retrieval | LLM Evaluation | Machine Learning | Model safety | Prompt engineering | Python | Red Teaming | Regression testing | Retrieval-Augmented Generation | SQL | Software testing | Test automation
Education
Roles
AI | AI QA | AI QA Trainer | Engineer | Evaluation Engineer | LLM Evaluation Engineer | QA Trainer | Trainer
Related jobs
-
Cloud Computing | Code review | Containerization | Docker | GPU ComputingFour-day work week | Fully remote | Open source contributions | Unlimited paid time offSenior-level Full TimeGermany R23h ago
-
AWS | Argo CD | Autoscaling | C# | CI/CD100 percent remote | Long-term contract | Part-time or full-timeMid-level Contract Full Time Part TimeBerlin R1d ago
-
AWS | Agent systems | Azure | Batch Processing | Cloud ArchitectureContract opportunity | Fully remote | Potential extension | Remote collaboration | US time zone overlapSenior-level Contract Full TimeGermany R1d ago
-
API Integration | Cloud Data | Cloud Data Warehouse | Data Contracts | Data LineageAutonomy in architecture | Collaborative environment | Flexible working arrangements | Fully remote work | High ownership roleSenior-level Full TimeGermany R1d ago
-
Junior Managers Program (Trainee) - Start your Leadership Journey in Business Intelligence & Analytics EUR 31K-34KBusiness Intelligence | Data Mining | Data Science | Data Visualization | Data analyticsFlexible work arrangements | Hybrid working | Mentoring | Professional development budget | SponsoringEntry-level Full TimeGerlingen, BW, Germany R1d ago
-
Junior Managers Program (Trainee) - Start your Leadership Journey in HR tech, digitalization and GenAI EUR 31K-34KArtificial Intelligence | Chatbot Development | Data analytics | English language | Generative AIFlexible working hours | Hybrid work model | Mentoring | Professional development budget | SponsoringEntry-level Full TimeGerlingen, BW, Germany R1d ago
-
(Senior) AI Engineer (all genders) EUR 65K-80KCloud Computing | Containerization | DevOps | Docker | Inference30 days vacation | Additional monitors | Businessbike | Company events | E-learning reimbursementMid-level Full TimeBremen, Munich, Mannheim, Mainz, Berlin, Remote R1d ago
-
A/B | A/B Testing | B testing | Contextual Bandits | Customer DataAnnual leave policy | Global team collaboration | In-person onboarding | Remote work | Restricted stock units (RSUs)Senior-level Full TimeGermany (Remote) R2d ago
-
Staff Data Scientist, Growth Data Products EUR 57K-74KA/B | A/B Testing | AWS | Airflow | Apache SparkDiscounts on company activities | Fitness budget | Health and wellness benefits | Hybrid work arrangement | Language reimbursementSenior-level Full TimeBerlin R2d ago
-
AI Agents | AI FinOps | AI systems | Access Control | Audit LoggingAccess to corporate events | Fully remote work | Medical insurance | Mental health support | Monthly benefits budgetExecutive-level Full TimeGermany R2d ago
-
API Design | API Gateways | Backend architecture | Best practices | Command LineCompany-provided equipment | Continuous learning allowance | Flexible working hours | Fully remote | Health insurance allowanceSenior-level Full TimeGermany R3d ago
-
AWS | Autoscaling | Azure | Bash | CI/CDFully remote work Europe | Healthcare coverage | Paid parental leave | Relocation assistanceMid-level Full TimeGermany R4d ago
-
Artificial Intelligence | Cloud Computing | Data Science | Generative AI | Machine Learning30 vacation days | Direct onboarding contact person | Discounted sports offers | Flat hierarchies | Flexible work hoursEntry-level Full TimeUlm, Germany R4d ago
-
Data Engineer, Vice President GBP 100K-130KAWS | Apache Airflow | Azure | CI/CD | Container OrchestrationEducation reimbursement | Family support programs | Flexible time off | Health and wellbeing resources | Hybrid work modelExecutive-level Full TimeL14 - Verde Building, 10 Bressenden … R4d ago
-
Senior Data Engineer GBP 91K-124KAccess Control | Airflow | BigQuery | CI/CD | Census401k employer match | Hackathons | Health and wellness benefits | Learning and development budget | Open source support fundSenior-level Full TimeGermany R4d ago
-
Apache Airflow | Azure | Azure Synapse | Azure Synapse Analytics | DBTFlexible working hours | Remote workSenior-level Full TimeGermany, Germany R4d ago
-
Agents | Data Architecture | Data Ingestion | Data Modeling | Data PipelinesAnnual performance bonus | Equipment support | Flexible working hours | Gifts and rewards | In-person team eventsSenior-level Full TimeGermany R4d ago
-
Senior AI Engineer (m/w/d) EUR 60K-68KAI Agents | AWS | Cloud Native | Code review | EvalsCompany events | Development budget | Ergonomic workstation | Family-friendly working hours | Fitness membership discountMid-level Full TimeRemote / Düsseldorf R4d ago
-
Senior AI Engineer (m/w/d) EUR 60K-68KAI Safety | AWS | Agentic Workflows | Artificial Intelligence | CI/CDAssistance with transportation | Development budget | Employer pension contributions | Ergonomic workplace | Events and partiesMid-level Full TimeRemote / Düsseldorf R4d ago
-
Alerting | Amazon Redshift | CI/CD | Dashboards | Data LineageCareer growth opportunities | Cross-functional collaboration | Fully remote work | Remote collaborationSenior-level Full TimeGermany R4d ago
-
Senior Data Engineer EUR 70K-90KCI/CD | ClickHouse | Docker | Elasticsearch | KafkaFlexible working hours | Learning time | Location Based Social Benefits | Remote-first | Team eventsSenior-level Full TimeGermany /Spain / Romania (Remote) R5d ago
-
Lead Software Engineer (AI/ML) EUR 56K-79KAPIs | AWS | Agent Orchestration | Autogen | AzureCompany hackathons | Company ticket | Flexible working hours | German language course | Hybrid work modelSenior-level Full TimeBerlin, Germany R5d ago
-
Azure | Data Lakehouse | Data Modeling | Data Pipelines | Data WarehousingCompany health management | Flexible working hours | Home office options | Hybrid work model | Jobrad eBike leasingSenior-level Full Time Part TimeRatingen, Germany; Ratingen, NRW, Germany; Ratingen R5d ago
-
Staff/Lead Python Engineer (FastAPI, Orchestration) EUR 82K-112KAsynchronous programming | Cloud platform | Concurrent programming | FastAPI | GenAIAnnual learning and development budget | Co-working budget | Company laptop | Enhanced parental leave | Flexible scheduleSenior-level Full TimeRemote Anywhere R6d ago
-
AI Engineer (gn) EUR 53K-53KAPI Integration | Access Control | Agents | By Design | DSGVOCompany pension | Employee discounts | Fitness subsidy | Flexible work hours | HomeofficeSenior-level Full TimeDeutschland, remote R7d ago