AI QA Trainer - LLM Evaluation
Tasks
- Analyze model behavior
- Assess hallucination detection
- Automate testing with Python and SQL
- Create evaluation frameworks
- Design test plans
- Develop evaluation rubrics
- Document findings and root cause analysis
- Evaluate large language models
- Execute regression suites
- Identify model failures
- Improve evaluation metrics and quality monitoring
- Perform red teaming
- Provide recommendations for prompts and guardrails
- Run adversarial testing
- Validate tool usage accuracy
- Verify grounding and retrieval augmented generation outputs
Perks/Benefits
Skills/Tech-stack
Adversarial Testing | Bias Assessment | Experiment tracking | Factual consistency | Hallucination detection | LLM Evaluation | Machine Learning | Prompt engineering | Python | Quality Assurance | RAG | Red Teaming | Regression testing | Retrieval-Augmented Generation | Robustness Testing | SQL | Safety testing | Software testing | Test Planning | Test automation
Education
Roles
AI | AI QA | AI QA Trainer | AI Quality Assurance Engineer | Engineer | QA Trainer | Quality Assurance Engineer | Trainer
Related jobs
-
AWS | Agent systems | Cloud platform | Compliance | Document ingestionContract engagement | Flexible consulting environment | Fully remote | US time zone overlapSenior-level Contract Full TimeItaly R1d ago
-
API Design | Artificial Intelligence | Channels | Concurrency | Distributed SystemsFlexible work location | Four-day work week | Fully remote work | Professional growth opportunities | Uncapped holiday allowanceSenior-level Full TimeItaly R1d ago
-
Data Engineer EUR 40K-60K.NET | Azure Cognitive | Azure Cognitive Services | Azure Data | Azure Data FactoryContinuous learning opportunities | Flexible working hours | Global team collaboration | Permanent contract | Remote work up to 3 days per weekSenior-level Full TimeReggio Emilia, Emilia - Romagna, Italy R2d ago
-
API Design | Command Line | Command-line interfaces | Data Modeling | Database systemsCompany-provided equipment | Continuous learning allowance | Flexible working hours | Fully remote work | Health insurance allowanceSenior-level Full TimeItaly R3d ago
-
Project Work in Clinical Data Science EUR 28K-28KData Mapping | Data Transformation | Data integration | Language Models | Language ProcessingEntry-level Full TimeITA - Milan - Trinita, Italy R3d ago
-
A/B | A/B Testing | AWS | Alerting | AutoscalingFully remote within Europe | Healthcare coverage | Paid parental leave | Professional growth | Relocation assistanceMid-level Full TimeItaly R4d ago
-
Individual Giving Data Engineer EUR 35K-39KAmazon Web Services | Apache Airflow | Apache Kafka | Cloud platform | DBTFully remoteMid-level Full TimeRome, Italy R4d ago
-
Alerting | Amazon Aurora | Amazon Aurora PostgreSQL | Amazon Redshift | Aurora PostgreSQLCross-functional collaboration | Exposure to modern data validation frameworks | Fully remote work | Ownership over data reliability strategySenior-level Full TimeItaly R4d ago
-
Engineering - Internal AI Transformation USD 74K-110KAPIs | Agentic Workflows | Auditability | Authentication | DockerAnnual company offsite | Co-working stipend | Learning and development stipend | Social travel stipendEntry-level Full TimeUnited States R8d ago
-
Auto Scaling | Blue-Green Deployment | Blue/green | CI/CD | CUDAFlexible remote work | Fully remote | Growth opportunities | High ownership and influence | Meaningful technical impactSenior-level Full TimeItaly R8d ago
-
Auto Scaling | CI/CD | CUDA | Container Orchestration | Distributed SystemsEnglish language support | On-call support | Remote workSenior-level Full TimeUkraine R14d ago
-
APIs | Agent systems | Autogen | Cloud services | CrewAIIncome protection | Paid time off | Pension match | Private healthcare | Remote work flexibilityMid-level Full TimeItaly R18d ago
-
AI/ML Engineer - Live Story EUR 35K-45KDeep learning | Hugging Face | LLM orchestration | Langchain | MLOpsFull remote | Milano office access | Remote-firstSenior-level Full TimeItaly - Remote R26d ago
-
Software / Machine Learning Engineer EUR 42K-57KComputer Vision | Data analytics | Deep learning | Generative AI | Hugging FaceFlexible work schedule | Gym discounts | Learning resources | Mental health support | MentorshipMid-level Full TimeMilan R1mo ago
-
Consultant - Quantitative Risk Consulting EUR 28K-28KAnomaly Detection | Artificial Intelligence | Basel | Business Intelligence | CosoApprenticeship contract | Flexible benefits | Health assistance | Hybrid work | Insurance coverageMid-level Full TimeRome - Villa Grazio, Italy R1mo ago
-
Junior Data Engineer EUR 19K-20KApache Airflow | Apache Spark | Data Governance | Data Modeling | Data PipelinesGym discounts | Learning resources | Mental health support | Mentorship program | Private healthcareEntry-level Full TimeMilan R1mo ago
-
Data Engineer EUR 40K-45KAWS | Apache Airflow | Apache Kafka | Apache Spark | CI/CDAdditional country specific advantages | Company events | Ergonomic setup | Flexible hybrid work model | Full remote availableMid-level Full TimeRome, Italy R1mo ago
-
Sr AI Engineer | Remote - Europe | TS/Vue/NodeJS GBP 80K-120KAPI Design | Agent architecture | Anthropic | Chat completion | Chunking401k match | Course budget | Dental coverage | Equity | HackathonsSenior-level Full TimeBerlin Office R1mo ago