AI QA Specialist (Conversational & LLM Systems)
Tasks
- Create and manage versioned datasets
- Define evaluation strategies for AI and LLM systems
- Define quality metrics and thresholds
- Detect regressions and evaluation drift
- Integrate evaluation practices into CI CD
- Provide quality signals for release decisions
- Validate conversational behavior end to end
- Verify intent matching and slot extraction
- Verify state transitions and business rules
- Verify tool and function calling correctness
Perks/Benefits
Skills/Tech-stack
Accuracy | CI/CD | Data Design | Data Quality | F1-score | Function Calling | Intent Recognition | JSON | JSON Schema | Language Models | Language Processing | Large Language Models | Monitoring | NLU | Natural Language | Natural Language Processing | Observability | Precisión | Python | Recall | Slot Extraction | Test Data | Test data design | Tool-Calling | Version control
Education
N/A
Roles
AI | AI QA | AI QA Specialist | Engineer | Machine Learning QA Engineer | QA Engineer | QA Specialist | Specialist
Related jobs
-
Data Quality Analyst ID52038 EUR 29K-40KAgile | CONCATENATE | COUNTIF | CSV | Data ValidationFlexible schedule | Mentorship | Office option | Personalized growth roadmaps | Remote work optionEntry-level Full TimeLisboa, Portugal3h ago
-
Data Quality Analyst ID52038 EUR 29K-40KAgile | CSV | EDI | Excel | JSONFlexible schedule | Mentorship | Office options | Personalized growth roadmaps | Remote work optionsEntry-level Full TimeCoimbra, Portugal3h ago
-
Data Quality Analyst ID52038 EUR 29K-40KAgile Development | Agile development methodology | CONCATENATE | COUNTIF | CSVEducation budget | Fitness budget | Flexible schedule | Mentorship | Office optionsEntry-level Full TimeAveiro, Portugal4h ago
-
Data Quality Analyst ID52038 EUR 28K-40KAgile | CSV | EDI | Excel | JSONEducation budget | Fitness budget | Flexible schedule | Mentorship | Office work optionsEntry-level Full TimePorto, Portugal4h ago
-
Senior-level Full TimePortugal, Lisbon, Portugal4h ago
-
Senior Data Engineer – AI & Agentic Pipelines EUR 38K-63KAWS | Agile | Amazon Redshift | Apache Airflow | CI/CDFlexible working hours | Hybrid work model | Private health insurance | Training and development | Volunteer daySenior-level Full TimeLisbon, Portugal6h ago
-
AI Solutions Architect EUR 59K-77K.NET | AI integration | API | API Development | Data ArchitectureSenior-level Full TimeLisbon, Lisboa, Portugal1d ago
-
Amazon Web Services | Apache Spark | Deep learning | Google Cloud | HadoopSenior-level Full TimeLisbon, Portugal1d ago
-
Senior Data Engineer EUR 55K-69KAirflow | Amazon S3 | ClickHouse | Data Lake | Data WarehouseCo-working | Company paid sick leave | English learning courses | Gym or swimming pool | Home office setup assistanceSenior-level Full TimePortugal - Remote R1d ago
-
Lead AI Engineer (LLMs & Data Pipelines) EUR 55K-78KEdge Computing | Embedded Systems | Embeddings | Inference Optimization | Language ModelsCareer path management | Flexible work culture | Performance-driven growth | Training and certificationSenior-level Full TimeLisbon, Portugal1d ago
-
AI Engineer (Speech & NLP Systems) EUR 54K-84KAutomatic Speech Recognition | Dataset curation | Deep learning | Edge AI | Embedded Machine LearningCareer path management | Fairness and Equity Salary Structure | Tech Startup Environment | Training and certificationSenior-level Full TimeLisbon, Portugal1d ago
-
Apache Kafka | Azure Data | Azure Data Factory | CI/CD | DAXBenefits plan | Hybrid work model | Remote work whenever possible | Work equipment providedSenior-level Full TimeLisbon, Portugal1d ago
-
Senior AI Research Scientist EUR 45K-66KArtificial Intelligence | Deep learning | Fine Tuning | HuggingFace Transformers | Language ModelsMid-level Full TimeLisbon, Lisboa, Portugal1d ago
-
Senior AI Engineer EUR 54K-81KAWS | Azure | Deep learning | Google Cloud | HadoopDiscounts | Flexible benefits | Health insurance | Meal allowance | Paid vacationSenior-level Full TimeLisbon, Portugal1d ago
-
Airflow | Apache Flink | Apache Hudi | Apache Iceberg | Apache KafkaFull remote | Healthcare industry focusSenior-level Full TimePorto, Portugal R1d ago
-
Senior AI Engineer - Hybrid Lisbon EUR 64K-93KAgile | Amazon Web Services | Apache Hadoop | Apache Spark | CI/CDEnglish B2 requirement | Hybrid workSenior-level Full TimeLisbon, Portugal R1d ago
-
Ansible | CI/CD | Docker | Git | JenkinsMid-level Full TimePorto, Portugal1d ago
-
Amazon Web Services | Apache Flink | Apache Kafka | Apache Spark | CassandraHybrid workSenior-level Full TimePorto, Portugal R1d ago
-
API Design | Agentic architecture | CI/CD | Cloud Computing | DockerEmployee stock option plan | Flexible scheduling | Fully remote | Health insurance | Home office setup allowanceMid-level Full TimePortugal R1d ago
-
Mid-level Full TimeLisbon, Lisbon, Portugal2d ago
-
Mid-level Full TimeLisbon, Lisbon, Portugal2d ago
-
Forward Deployed Engineer (all genders) EUR 45K-67KABAC | AI Agents | API Integration | Amazon Web Services | Async ProgrammingMid-level Full TimeLisbon, Lisbon, Portugal2d ago
-
Embedded Systems Engineer Innovation EUR 36K-47KARM Cortex | ARM Cortex-M | Analog electronics | Arduino | Bluetooth Low EnergyMid-level Full TimeSão João da Madeira, Portugal, Portugal2d ago
-
API Integration | Amazon Redshift | Bitbucket | CI/CD | Data GovernanceAgile team environment | End of week summaries | Remote work | Weekly update callsMid-level Full TimePortugal R2d ago
-
Data Engineer Expert EUR 42K-65KAnalysis Services | Apache Spark | Azure Analysis | Azure Analysis Services | Azure DataFlexible benefits | Health insurance | Hybrid work | Meal allowance | Paid vacationSenior-level Full TimeLisbon, Portugal2d ago