AI QA Trainer - LLM Evaluation
Tasks
- Analyze model failures
- Assess hallucination detection
- Automate evaluation using Python
- Automate evaluation using SQL
- Collaborate on evaluation metrics and dashboards
- Conduct red teaming
- Create evaluation frameworks
- Define pass fail criteria
- Design test plans
- Develop evaluation rubrics
- Document failure reports
- Evaluate large language models
- Evaluate retrieval augmented generation outputs
- Execute regression suites
- Maintain evaluation documentation
- Monitor model quality
- Perform adversarial testing
- Perform root cause analysis
- Recommend guardrails improvements
- Recommend prompt improvements
- Run robustness evaluations
- Support test automation
- Validate tool usage accuracy
- Verify grounding
Perks/Benefits
- Contract position
- Flexible work hours
- High-speed internet required
- Own computer required
- Remote work
Skills/Tech-stack
AI Safety | AI Testing | AI testing frameworks | Adversarial Testing | Bias Assessment | Experiment tracking | Factual consistency | Hallucination detection | LLM Evaluation | Machine Learning | Prompt engineering | Python | Red Teaming | Regression testing | Retrieval-Augmented Generation | SQL | Safety testing | Test automation | Testing Frameworks
Education
Roles
AI | AI QA | AI QA Trainer | Evaluation Specialist | LLM Evaluation Specialist | QA Trainer | Specialist | Trainer
Related jobs
-
Gen AI Engineer CAD 126K-170KAWS | Amazon Bedrock | CI/CD | CrewAI | DevOpsCareer development opportunitiesSenior-level Full TimeCanada - Remote R1d ago
-
AI orchestration | AWS | Agent systems | Conversation memory | Document processingContract extension opportunity | Fully remote | US time zone overlapSenior-level Contract Full TimeCanada R1d ago
-
Senior-level Full TimeCanada - Remote R2d ago
-
AI Developer CAD 100K-160KAI Agents | AI Pipelines | AI Services | API Integration | LLM APIsBeverages provided at office | Commute benefit contribution | Employee referral program | Fitness reimbursement contribution | Flexible scheduleMid-level Full TimeMontréal, QC (Hybride) R2d ago
-
AI Architect (T & I) (Telework/Hybrid) CAD 110K-130KA2A | AI Agent | AI agent architectures | AWS | AccelerateHybrid work | TeleworkSenior-level Full TimeMontreal - MRC (Papineau) (36.25), Canada R3d ago
-
AI Engineer Intern CAD 40K-60KAI guardrails | API Development | APIs | Cost Optimization | DatabasesDowntown office location | Equal opportunity employment | Hybrid work | Learning and development resourcesEntry-level InternshipToronto, Ontario R3d ago
-
AI Foundry | Agent systems | Anthropic Claude | Azure AI | Azure AI FoundryCaregiver leave | Continuous development support program | Employee assistance program | Flexible paid time off | Parental leaveMid-level Full TimeMississauga R3d ago
-
Dashboard Development | Data Analysis | Data Quality | Data Studio | Data VisualizationCross-functional collaboration | Flexible schedule | Fully remote work | Inclusive workplace culture | International projectsMid-level Full TimeCanada R3d ago
-
AI Engineer, Marketing AI Automation CAD 67K-90KCloud Computing | Full Stack | Full-Stack Development | Git | JavaScriptSenior-level Full TimeCAD Remote – ON, Canada R4d ago
-
AI Solutions Engineer CAD 90K-110KAI Agents | API Integration | Alerting | Cause analysis | Content ManagementDental insurance | Employee discounts | Medical insurance | Paid time off | RRSP matchMid-level Full TimeHybrid- Toronto, ON R4d ago
-
Airflow | Autogen | ClickHouse | CrewAI | DagsterAnnual performance bonus | Equipment support | Flexible working hours | Gifts and rewards | Lunch allowanceSenior-level Full TimeCanada R4d ago
-
AI/ML Technical Practice Lead CAD 131K-170KAWS Agentic AI | AWS CloudFormation | AWS ECS | AWS Fargate | AWS IAMContinuous learning culture | Professional development | Remote work | Training sessions | Travel opportunitiesSenior-level Full TimeCanada - Remote R5d ago
-
AI Data Engineer CAD 120K-154KAWS | Amazon Redshift | CI/CD | Data Governance | Data ModelingCritical illness insurance | Extended Healthcare Plan | Flexible hours | Group Life AD D | Group RRSPSenior-level Full TimeToronto, Ontario, Canada - Remote R5d ago
-
AWS | Apache Airflow | CI/CD | Cloud platform | Data PipelinesAccess to modern development tools and hardware | Collaborative innovation focused environment | Continuous learning opportunities | Equity opportunities | Medical, dental, and vision benefitsMid-level Full TimeCanada R7d ago
-
Amazon Web Services | Apache Spark | Big Data | Data Analysis | Data setsRemote workSenior-level Full TimeMontreal, QC, Canada R7d ago
-
Engineering - Internal AI Transformation USD 74K-110KAPIs | Agentic Workflows | Auditability | Authentication | DockerAnnual company offsite | Co-working stipend | Learning and development stipend | Social travel stipendEntry-level Full TimeUnited States R8d ago
-
Product & AI Intern 4 Months - Fall 2026 CAD 56K-64KConfluence | Databricks | Language Models | Language Processing | Large Language ModelsDevelopment programs | Recognition programs | Time away from work programs | Volunteer days | Wellness initiativesEntry-level InternshipCanada R8d ago
-
Full-Stack AI Developer Intern CAD 52K-62KData Analysis | Docker | JavaScript | Language Models | Large Language ModelsRemote workEntry-level Full Time InternshipCAD Remote – ON, Canada R10d ago
-
Anthropic Claude | Anthropic Claude API | Claude API | Embeddings | Evaluation harnessesEntry-level Contract InternshipEtobicoke, ON - Remote R12d ago
-
API Integration | Asynchronous programming | Automation | Data Pipelines | Data TransformationEntry-level Contract InternshipEtobicoke, ON - Remote R12d ago
-
AI Solutions Engineer USD 127K-204KAPI Integrations | Anthropic Claude | Benefit Analysis | Change Management | Cost BenefitCareer development | Distributed team collaboration | Learning opportunities | Work from anywhereMid-level Full TimeCanada R14d ago
-
AI Engineering Lead, Product Analytics CAD 140K-175KAI Governance | AI compliance | Agent Design | Azure DevOps | Data LakeCareer development | Employee incentive programs | Flexible vacation | Flexible work-life balance | Headspace accessSenior-level Full TimeCanada, Toronto, Ontario R19d ago
-
AWS SageMaker | Amazon Web Services | Automated testing | CI/CD | Distributed ComputingSenior-level Full TimeAMER - Canada - Quebec - … R22d ago
-
AI/ML Specialist Solutions Architect USD 220K-375KAWS | Ansible | Azure | CUDA | Cloud platform401k match | Career growth | Collaborative culture | Disability insurance | Equity participationSenior-level Full TimeCanada; Remote - United States R22d ago
-
AI Solutions Engineer CAD 173K-233KAPIs | Authentication | CI/CD | Circuit Breakers | ContainersDental insurance | Employee stock purchase plan | Flexible spending accounts | Health insurance coverage | Paid time offMid-level Full TimeRemote Canada R23d ago