AI QA Trainer - LLM Evaluation
Tasks
- Analyze model behavior and recommend prompt and guardrail improvements
- Automate testing and evaluations using scripting
- Collaborate on evaluation metrics dashboards and quality monitoring
- Design and execute test plans and regression suites
- Develop evaluation rubrics and quality benchmarks
- Evaluate large language models across quality and safety criteria
- Identify and document model failures and vulnerabilities
- Maintain documentation and reproducible issue reports
- Perform adversarial testing and red teaming
- Verify grounding retrieval augmented generation outputs and tool usage accuracy
Perks/Benefits
Skills/Tech-stack
Adversarial Testing | Bias Assessment | Experiment tracking | Factual consistency | Hallucination detection | LLM Evaluation | Machine Learning | Prompt engineering | Python | RAG | Red Teaming | Regression testing | Retrieval-Augmented Generation | SQL | Safety testing | Test automation
Education
Roles
AI | AI QA | AI QA Trainer | Engineer | Learning Engineer | Machine Learning Engineer | QA Trainer | Trainer
Related jobs
-
AWS | Agent systems | Artificial Intelligence | Azure | Cloud platformContract extension opportunity | Fully remote | US time zone overlapSenior-level Contract Full TimeIreland R1d ago
-
API Integration | Cloud Data | Cloud Data Warehouse | Data Contracts | Data IngestionCollaborative environment | Flexible working hours | Fully remote | High autonomy | Professional growth opportunitiesSenior-level Full TimeIreland R1d ago
-
A/B | A/B Testing | B testing | Data Pipelines | Data version controlAnnual leave policy | Global culture | Remote work | Restricted stock unitsSenior-level Full TimeRepublic of Ireland (Remote) R2d ago
-
API | API Design | Backend architecture | Command Line | Command-line InterfaceCompany-provided equipment | Continuous learning allowance | Flexible working hours | Fully remote | Health insurance allowanceSenior-level Full TimeIreland R3d ago
-
Lead, Finance Analytics & Enablement AI/ML EUR 50K-61KAmplitude | Business Intelligence | Cloud platform | Data Modeling | Data ScienceFlexdesk Access Pass | Travel stipend | Work from homeSenior-level Full TimeIreland - Remote R3d ago
-
A/B | A/B Testing | AWS | Alerting | AzureFully remote Europe | Healthcare coverage | Paid parental leave | Professional growth | Relocation assistanceMid-level Full TimeIreland R4d ago
-
Senior Data Engineer GBP 91K-124KAccess Control | Airflow | BigQuery | CI/CD | Census401k employer match | Hackathons | Health and wellness benefits | Learning and development budget | Open source support fundSenior-level Full TimeGermany R4d ago
-
Alerting | Amazon Aurora | Amazon Aurora PostgreSQL | Amazon Redshift | Anomaly DetectionContinuous improvement focus | Cross-functional collaboration | Exposure to modern data platforms | Fully remote work | Ownership opportunitiesSenior-level Full TimeIreland R4d ago
-
Auto Scaling | Blue-Green Deployment | Blue/green | CI/CD | CUDAFlexible work schedule | Fully remote work | Ownership and influence | Remote-first environment | Startup growth opportunitiesSenior-level Full TimeIreland R8d ago
-
Staff AI Engineer - 2nd Horizon | Ireland | Remote EUR 117K-147KAWS | Agent systems | Agentic Workflows | Cloud platform | Context engineeringAnnual leave policy | In-person onboarding | Remote workSenior-level Full TimeRepublic of Ireland (Remote) R8d ago
-
Lead Data Analytics & AI Engineer EUR 78K-117KAPI | AWS | Amazon Kinesis | Apache Airflow | Apache FlinkHealth insurance | Hybrid work | Long-term illness protection | Pension | Referral bonusSenior-level Full TimeIE Dublin, Ireland R10d ago
-
Data Platform Engineer EUR 45K-57KAirflow | ArgoCD | Avo | CI/CD | DBTCollaborative team | Hybrid work setupMid-level Full TimeAmsterdam; Dublin; Helsinki HQ R11d ago
-
Senior Data Analytics & AI Engineer EUR 65K-98KAPIs | AWS | Apache Airflow | CI/CD | DBTBonus plan | Health insurance | Hybrid work | Long-term illness protection | PensionSenior-level Full TimeIE Dublin, Ireland R16d ago
-
Senior Gen AI Developer USD 32K-32KAPI Integration | AWS | Agent Orchestration | Angular | Distributed SystemsHybrid work | Learning and development opportunities | Office based collaboration 3 days per week | Supportive leadershipSenior-level Full TimeDublin - Frame, Ireland R17d ago
-
Senior-level Full TimeIreland - Dublin - Hybrid R18d ago
-
AI/Machine Learning Engineer EUR 85K-90KAWS | CI/CD | Classification | Cloud platform | Code ManagementAnnual leave | Dental insurance | Employee assistance program | Maternity leave | Paternity leaveMid-level Full TimeDublin, County Dublin, Ireland - Remote R22d ago
-
Staff ML Engineer EUR 56K-80KAWS | Amazon SageMaker | Apache Airflow | Apache Kafka | CI/CDHome office stipend | Manager training | PTO | Paid parental leave | Professional development coursesSenior-level Full TimeIreland R24d ago
-
Senior Software Engineer, Analytics Platform EUR 56K-88KAWS | Ansible | Apache Druid | Apache Flink | Apache KafkaCareer development | Dental plan | Hybrid work options | Mentoring | Paid Maternity LeaveSenior-level Full TimeVirtual Office (Ireland) R25d ago
-
AI Software Engineer II (R-19524) EUR 42K-60KAPIs | Agentic Workflows | CI/CD | Container Orchestration | DebuggingAnnual leave | Educational assistance program | Employee Health Insurance | Family-friendly leave | Flexible working (hybrid model)Mid-level Full TimeDublin - Ireland R25d ago
-
Angular | Backlog Management | Databricks | JavaScript | MEAN StackHybrid work | Remote work flexibilitySenior-level Full TimeDublin - Frame, Ireland R29d ago
-
Senior Analytics Engineer (AI & Quality) EUR 44K-57KCloud platform | Data Governance | Data Modeling | Data Quality | GitHubInternational team collaboration | Remote work | Supportive work environmentSenior-level Full TimeDublin, Ireland R1mo ago
-
Machine Learning Engineer EUR 60K-76KAWS | Azure | Cloud infrastructure | Experiment tracking | GCPDonation support | Generous time off | Healthcare | Occasional travel | Parental leaveMid-level Full TimeRemote - Ireland R1mo ago
-
Causal Inference | Data Analysis | Decision Trees | Embeddings | Ensemble ModelsCaregiver reimbursement | Employee stock purchase plan | Flexible working hours | Hackathons | Learning groupsSenior-level Full TimeLeinster-Dublin, Ireland R1mo ago
-
Senior AI Engineer EUR 58K-86KArtificial Intelligence | CI/CD | Docker | ETL | KubernetesHealth insurance | Hybrid work | Professional development opportunities | Supportive leaders | Workplace wellnessSenior-level Full TimeDublin - Charlotte, Ireland R1mo ago
-
Sr AI Engineer | Remote - Europe | TS/Vue/NodeJS GBP 80K-120KAPI Design | Agent architecture | Anthropic | Chat completion | Chunking401k match | Course budget | Dental coverage | Equity | HackathonsSenior-level Full TimeBerlin Office R1mo ago