AI QA Trainer - LLM Evaluation
Tasks
- Analyze model failures and bias issues
- Assess safety and reliability
- Automate testing workflows
- Conduct adversarial testing and red teaming
- Create test plans and regression suites
- Design evaluation frameworks
- Develop evaluation rubrics and benchmarks
- Document findings and root cause analysis
- Evaluate large language models
- Perform hallucination detection testing
- Provide recommendations for prompts and guardrails
- Support quality monitoring dashboards
- Verify grounding and retrieval augmented generation outputs
Perks/Benefits
Skills/Tech-stack
Adversarial Testing | Bias Assessment | Factual consistency | Hallucination detection | LLM Evaluation | Language Models | Large Language Models | Machine Learning | Prompt engineering | Python | Red Teaming | Regression testing | Retrieval-Augmented Generation | Risk Assessment | SQL | Safety testing | Test automation
Education
Related jobs
-
AWS | Agent systems | Amazon Bedrock | Azure | ClaudeConsulting environment | Contract extension opportunity | Fully remote | Remote contract | US time zone overlapSenior-level Contract Full TimePortugal R1d ago
-
Senior Data Engineer USD 150K-200KAPIs | Amazon QuickSight | Amazon Web Services | Backup | CI/CDBusiness English classes | Continuous learning support | Defined career path | Every other Friday off | Explore programSenior-level Full TimeLatAm, North Macedonia, Portugal, Serbia R1d ago
-
Senior Machine Learning Engineer EUR 68K-89KA/B | A/B Testing | AWS | B testing | BentoMLCode review culture | Hybrid work model | Mentorship | Remote work flexibilitySenior-level Full TimePortugal - All - Fully Flexible R2d ago
-
AWS | Anthropic Claude | Anthropic Claude API | Apache Spark | Automated testing15th month salary | Birthday off | Family coverage | Health insurance | Mobility programSenior-level Full TimePorto, Portugal R2d ago
-
API Design | Backend architecture | Command Line | Command-line Interface | Database systemsCompany-provided equipment | Continuous learning allowance | Flexible working hours | Fully remote work | Health insurance allowanceSenior-level Full TimePortugal R3d ago
-
API Integration | Artificial Intelligence | Change Management | Data Privacy | Generative AIInclusive workplace | Innovation culture | Remote work | Travel for onsite collaborationExecutive-level ContractEurope R3d ago
-
Anthropic API | Async Programming | Autogen | CI/CD | Cloud Data15th month salary | Birthday off | Health insurance | Mobility program | Work-life balanceSenior-level Full TimePorto, Portugal R3d ago
-
AWS | Autoscaling | Azure | Bash | CI/CDFully remote within Europe | Healthcare coverage | Paid parental leave | Professional growth | Relocation assistanceMid-level Full TimePortugal R4d ago
-
Data Analyst & Administrator Trainee EUR 28K-28KDashboard Development | Data Analysis | Data Integrity | Data Visualization | Data cleaningCross-functional team | Flexible working hours | Hybrid work schedule | MentorshipEntry-level Full TimeLisboa, Portugal R4d ago
-
Senior Data Engineer GBP 91K-124KAccess Control | Airflow | BigQuery | CI/CD | Census401k employer match | Hackathons | Health and wellness benefits | Learning and development budget | Open source support fundSenior-level Full TimeGermany R4d ago
-
Senior AI Engineer - AI Systems EUR 48K-65KAnthropic API | Application engineering | Async | Autogen | CI/CDHealth insurance | Meal allowance | No-term contract | Paid vacation | Referral bonusSenior-level Full TimeLisbon, Portugal R4d ago
-
Forward Deployed AI Engineer EUR 65K-78KAI Agents | API Integration | AWS | Agentic Workflows | AutogenAnnual leave increased every 3 years | Continuous learning | Flexible benefits plan | Hybrid work | Meal allowanceMid-level Full TimeLisbon, Portugal R4d ago
-
Forward Deployed AI Engineer EUR 65K-78KAI Agents | API Integration | Amazon Web Services | Autogen | AzureFlexible benefits plan | Flexible hybrid work model | Meal allowance | Paid time off | Private medical insuranceMid-level Full TimeLisbon, Portugal R4d ago
-
Alerting | Amazon Aurora | Amazon Redshift | CI/CD | DashboardsContinuous improvement culture | Cross-functional collaboration | Exposure to modern data platforms | Fully remote work | Opportunity to influence engineering standardsSenior-level Full TimePortugal R4d ago
-
Azure | Data Drift | Data Modeling | Data Processing | Data QualityFlexible schedule | Freelance service agreement | Remote workMid-level Full TimePortugal - Remote R5d ago
-
Junior AI & Data Engineer EUR 18K-27KAgile | CI/CD | Docker | Kubernetes | Machine LearningCareer development | Discounts | Flexible work hours | Health insurance | Hybrid work modelEntry-level Full TimeLisboa, PT, 1200-246 R5d ago
-
A/B | A/B Testing | B testing | Chroma | EmbeddingsHybrid work modelSenior-level Full TimeLisbon, Portugal R5d ago
-
AWS | Azure | BERT | CI/CD | Deep learningBirthday off | Family coverage | Health insurance | Mobility program | Real work/life balanceEntry-level Full TimePorto, Portugal R5d ago
-
Staff/Lead Python Engineer (FastAPI, Orchestration) EUR 82K-112KAsynchronous programming | Cloud platform | Concurrent programming | FastAPI | GenAIAnnual learning and development budget | Co-working budget | Company laptop | Enhanced parental leave | Flexible scheduleSenior-level Full TimeRemote Anywhere R6d ago
-
Engineering - Internal AI Transformation USD 74K-110KAPIs | Agentic Workflows | Auditability | Authentication | DockerAnnual company offsite | Co-working stipend | Learning and development stipend | Social travel stipendEntry-level Full TimeUnited States R8d ago
-
Auto Scaling | Blue-Green Deployment | Blue/green | CI/CD | CUDACareer growth opportunities | Exposure to distributed systems and GPU computing | Fully remote flexibility | High ownership | Meaningful technical impactSenior-level Full TimePortugal R8d ago
-
APIs | Airtable | Analytics | Artificial Intelligence | AttributionAutonomy | Equity | Flexible hours | Fully remote | High ownershipMid-level Full TimePoland R8d ago
-
Machine Learning Engineer EUR 56K-86KA/B | A/B Testing | Alerting | B testing | BigQueryAgile working | Enhanced annual leave | Paid family leave | Paid volunteering days | Support for growth and developmentSenior-level Full TimePorto, Portugal R8d ago
-
Senior AI-First Data Engineer EUR 75K-125KApache Airflow | Business Intelligence | CRM | Customer Data | Customer data platformsRemote-friendlySenior-level Full TimeArmenia - Remote; Europe; Georgia - … R9d ago
-
Senior Data Engineer EUR 69K-103KAirflow | CI/CD | DBT | Data Modeling | Data QualityHybrid work model | Remote work flexibilitySenior-level Full TimePortugal - Lisbon - Hybrid R10d ago