AI QA Trainer - LLM Evaluation
A AUD 50K-100K (estimate) Senior-level Contract Full Time
Tasks
- Analyze model behavior and recommend improvements
- Assess tool usage accuracy and workflow reliability
- Automate evaluation using Python or SQL
- Collaborate on evaluation metrics dashboards and monitoring
- Design test plans and regression suites
- Develop evaluation rubrics and quality benchmarks
- Evaluate large language models
- Identify and document model failures
- Maintain reproducible documentation of findings
- Perform adversarial testing and red teaming
- Validate grounding and retrieval augmented generation outputs
- Verify hallucination detection and safety
Perks/Benefits
Skills/Tech-stack
AI Quality Assurance | AI quality | Adversarial Testing | Bias Assessment | Data Analysis | Evaluation Frameworks | Experiment tracking | Factual consistency | Hallucination detection | LLM Evaluation | Language Models | Large Language Models | Machine Learning | Prompt engineering | Python | Quality Assurance | Quality Benchmarks | RAG | Red Teaming | Regression testing | Retrieval-Augmented Generation | SQL | Safety testing | Software testing | Test automation | Tool usage
Education
Roles
AI | AI QA | AI QA Trainer | Evaluation Specialist | LLM Evaluation Specialist | QA Trainer | Specialist | Trainer
Related jobs
-
Solutions Architect AUD 132K-180KAPIs | Agentic AI | Cloud Computing | Computer Vision | Container OrchestrationRemote work flexibility | Training sessionsSenior-level Full TimeRemote - Australia R1d ago
-
AWS | Agent systems | Cloud Architecture | Cloud platform | ComplianceFlexible consulting environment | Fully remote | US time zone overlapSenior-level Contract Full TimeAustralia R1d ago
-
Amazon Web Services | Apache Spark | Big Data | Data Analysis | HadoopRemote workSenior-level Full TimeSydney, NSW, Australia R7d ago
-
Forward Deployed AI Engineer AUD 148K-180KAWS Bedrock | Apex | CI/CD | Cloud platform | EvalsFlexible working arrangements | Home office expense allowance | MuleSoft certifications paid | Paid parental leave | Project management certifications paidSenior-level Full TimeSydney, Australia R15d ago
-
AI / Data Platform Engineer AUD 100K-127KAPI Design | Azure | Azure OpenAI | BI Dataflows | BI Semantic ModelAdditional leave benefits | Annual volunteer days | Birthday leave | Discounts on products | Dog-friendly officesEntry-level Full TimeBrisbane, Queensland (AU) - Bell-Are Ave, … R19d ago
-
AI Engineer - Melbourne AUD 92K-125KAPI Development | AWS | Automated testing | CI/CD | Cloud platformMid-level Full TimeMelbourne - 727 Collins, Australia R26d ago
-
Mid-level Full TimeSydney - Barangaroo, Australia R26d ago