AI QA Trainer - LLM Evaluation
Tasks
- Analyze model behavior and recommend improvements
- Automate testing using Python or SQL
- Conduct robustness evaluations
- Design test plans and evaluation frameworks
- Develop evaluation rubrics and quality benchmarks
- Document findings and root cause analysis
- Evaluate large language models
- Execute regression suites
- Identify model failures and vulnerabilities
- Improve AI evaluation metrics dashboards and quality monitoring
- Perform adversarial testing and red teaming
- Verify grounding and retrieval augmented generation outputs
- Verify tool usage accuracy
Perks/Benefits
Skills/Tech-stack
Adversarial Testing | Artificial Intelligence | Bias Assessment | Experiment tracking | Hallucination detection | LLM Evaluation | Language Models | Large Language Models | Machine Learning | Prompt engineering | Python | Quality Assurance | RAG | Red Teaming | Regression testing | Retrieval-Augmented Generation | SQL | Safety testing | Software testing | Test automation
Education
Roles
AI | AI QA | AI QA Engineer | Engineer | Learning Engineer | Machine Learning Engineer | QA Engineer
Related jobs
-
Cloud infrastructure | Containerization | Docker | GPU Computing | KubernetesCareer growth in AI infrastructure | Four-day work week | Fully remote | Global team collaboration | Open communication inclusive cultureSenior-level Full TimeBrazil R22h ago
-
AWS Textract | Autogen | Azure Form Recognizer | CI/CD | Computer VisionBirthday day off | Childcare assistance | Cultural partnerships | Dental insurance | Family support for new parentsSenior-level Full TimeBrazil R1d ago
-
AI Governance | AWS | Agent systems | Azure | Cloud ArchitectureContract extension opportunity | Flexible consulting environment | Fully remote | US time zone overlapSenior-level Contract Full TimeBrazil R1d ago
-
AI Foundry | API Management | AWS Bedrock | Active Directory | Application InsightsAccess to multicultural global projects | Birthday day off | Development tracks | Equipment provided | Fitness programsSenior-level Full TimeBrazil R1d ago
-
AWS Glue | Apache Airflow | Apache Kafka | Apache Spark | CassandraCollaborative environment | Continuous learning and knowledge sharing | Fully remote work model | Inclusive workplace open to diverse backgrounds | Professional growth opportunitiesSenior-level Full TimeBrazil R2d ago
-
Senior Applied AI Software Engineer BRL 300K-360KAI orchestration | API Development | AWS | Agentic Workflows | Automated testingRemote work | Work in US Eastern time zoneSenior-level Full TimeSão Paulo, SP, Brazil R2d ago
-
Agile | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake StorageCompany-sponsored certifications | Continuing education | Flexible working hours | Paid time off | Remote workSenior-level Full TimePorto Alegre, Brazil R2d ago
-
Agile | Azure | Azure Data | Azure Data Factory | Azure Data LakeCompany-sponsored certifications | Continuing education | Flexible working hours | Paid time off | Remote workSenior-level Full TimeSão Paulo, Brazil R2d ago
-
Agile | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake StorageCompany-sponsored certifications | Continuing education | Flexible working hours | Paid time off | Remote workSenior-level Full TimeSalvador, Bahia, Brazil R2d ago
-
Agile | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake StorageCertification support | Company-sponsored coursework | Conference attendance | Continuing education | Flexible working hoursSenior-level Full TimeRio de Janeiro, Brazil R2d ago
-
Agile | Azure | Azure Data | Azure Data Factory | Azure Data LakeCompany-sponsored certifications | Conference attendance | Continuing education | Flexible working hours | Paid time offSenior-level Full TimeRecife, Pernambuco, Brazil R2d ago
-
Agile | Azure | Azure Data | Azure Data Factory | Azure Data LakeCompany-sponsored certifications | Conference support | Continuing education | Flexible working hours | Paid time offSenior-level Full TimeManaus, Amazonas, Brazil R2d ago
-
Azure | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake StorageCompany-sponsored certifications | Conferences | Continuing education | Flexible working hours | Paid time offSenior-level Full TimeGoiânia, Goiás, Brazil R2d ago
-
Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure DevOpsCompany-sponsored certifications | Company-sponsored coursework | Conferences | Continuing education | Flexible working hoursSenior-level Full TimeFortaleza, Ceará, Brazil R2d ago
-
Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure DevOpsCompany sponsored certifications exams coursework | Conference opportunities | Continuing education | Flexible working hours | Paid time offSenior-level Full TimeCuritiba, Paraná, Brazil R2d ago
-
Azure | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake StorageCompany-sponsored certifications | Company-sponsored coursework | Continuing education | Flexible working hours | Paid time offSenior-level Full TimeBrasília, Federal District, Brazil R2d ago
-
Access Controls | Agile | Azure Data | Azure Data Factory | Azure Data LakeCompany-sponsored certifications | Conference attendance | Continuing education | Flexible working hours | Paid time offSenior-level Full TimeBelo Horizonte, Minas Gerais, Brazil R2d ago
-
Artificial Intelligence | Artificial Intelligence Reasoning | Attention to detail | Content Moderation | Data AnnotationFreelance project | Part-time | Remote workEntry-level FreelanceBrazil - Remote R2d ago
-
Senior-level Full TimeCuritiba (Hybrid), Brazil R2d ago
-
FBS Associate Analytics Engineer (Remote) BRL 150K-220KAmazon Web Services | Apache Airflow | Cloud platform | DBT | Google CloudFlexible work arrangements | Inclusive work culture | Paid time off | Private health insurance | Remote workMid-level Full TimeBrazil - Remote R2d ago
-
Senior Data Engineer (Databricks/AWS) BRL 319K-443KAWS | Amazon S3 | Apache Airflow | CI/CD | Cluster tuningRemote workSenior-level ContractBrazil - Remote R2d ago
-
AI Engineering I - Marketing USD 30K-39KAWS | BERT | CI/CD | Docker | EmbeddingsFull-time employment | Remote workEntry-level Full TimeBrazil, Remote R2d ago
-
AWS | Amazon S3 | DBT | Dagster | ETLFully remote | Learning oriented work environment | Opportunity for contract extension | Technical ownership cultureSenior-level Full TimeBrazil R2d ago
-
Anthropic | Benchmarking | CI/CD | Docker | FastAPIAutonomy | Fully remote | PJ contract structureSenior-level Full TimeBrazil R3d ago
-
API documentation | Airflow | Amazon Web Services | Apache Beam | Apache SparkBirthday day off | Dental insurance | Educational support programs | Fitness and wellness partnerships | Fully remote workMid-level Full TimeBrazil R3d ago