Senior AI Quality Engineer (LLM Evaluation & Automation) 1754
Tasks
- Build eval harness
- Create regression packs
- Define scorecard metrics
- Develop adversarial testing plan
- Develop drift testing plan
- Fail builds on quality regression
- Generate candidate eval cases with AI
- Integrate evals into CI
- Maintain exception tasks
- Maintain golden tasks
- Partner with product and tech lead on thresholds
- Set release gate thresholds
Perks/Benefits
Skills/Tech-stack
Automation | Benchmarking | CI/CD | Case design | LLM Evaluation | Language Models | Large Language Models | Machine Learning | Probabilistic Modeling | Regression testing | Scripting | Test Case | Test Case Design
Education
N/A
Related jobs
-
Lead AI Engineer USD 160K-213KAgentic Workflows | Artificial Intelligence | CI/CD | Data Engineering | LLM ApplicationsHybrid work model | In-office collaboration | Remote flexibilitySenior-level Full TimeTel Aviv-Yafo, Tel Aviv District, IL R7h ago
-
Alerting | Amazon Redshift | Aurora PostgreSQL | CI/CD | Cause analysisCollaborative environment | Fully remote positionSenior-level Full TimeBARRIO ARCA, Argentina R7h ago
-
Sr Data Engineer - Pessoa com deficiência (PCD) BRL 54K-60KAPIs | Batch Processing | BigQuery | Data pipeline | GitCareer growth | Hybrid work | Learning opportunities | Team collaboration | Workplace flexibilitySenior-level Full TimeSudeste,Brazil R8h ago
-
Mid-level Full TimePetah Tikva, Center District, IL R8h ago
-
Agentic AI | Alerting | Anomaly Detection | Apache Airflow | Artifact RegistryMid-level Full TimePune, Maharashtra, India R8h ago
-
Senior Data Engineer INR 3000K-4000KAWS | Airflow | Amazon Aurora | Apache Kafka | Apache SparkBike to work | Career mentorship | Educational assistance | Employee assistance program | Employee discountSenior-level Full TimeMumbai, MH, India R9h ago
-
Ads ML Developer Experience Tech Lead USD 174K-220KAI Agents | Agent Orchestration | Artificial Intelligence | Bias Mitigation | Build systemsSenior-level Full TimeSunnyvale, CA R10h ago
-
Azure Data Engineer INR 1500K-2500KAzure Data | Azure Data Factory | Azure Data Lake | Azure Functions | Azure SynapseSenior-level Full TimeHyderabad, Telangana, India R10h ago
-
Data Platform Engineer GBP 55K-75KApache Flink | Apache Kafka | Automation | BigQuery | Cloud SpannerHybrid work from homeEntry-level Full TimeStoke-on-Trent, England, United Kingdom R11h ago
-
AI/ML Engineer (Synthetic Data) BE CZK 960K-1260KAWS | CI/CD | CatBoost | Claude Code | DatabricksCompany laptop | Meal vouchers | Paid vacation | Pension fund contribution | Remote work optionMid-level Full TimePrague, Prague, Czechia R11h ago
-
Senior DevOps Engineer ILS 283K-366KAWS | Argo CD | BigQuery | CI/CD | ClearMLHybrid work environment | Inclusive workplaceSenior-level Full TimeRamat Gan, Tel Aviv District, IL R12h ago
-
Data Engineer UAH 60K-88KCI/CD | Docker | Git | Graph Databases | Integration TestingCompany-paid certifications | Conference and workshop support | Financial consultations | Language courses | Legal consultationsMid-level Full TimeUkraine R12h ago
-
Senior Infrastructure and Data Engineer (Remote in Europe) DKK 800K-1201KAlerting | CI/CD | Cloud platform | Data Normalization | Data PipelinesCET compatible time zone | Full ownership | Remote workSenior-level Full TimeDenmark - Remote R13h ago
-
AI Search | Application Insights | Azure | Azure AI | Azure AI SearchMid-level Full TimeNew York, United States R14h ago
-
Senior Machine Learning Engineer AUD 144K-180KAWS | Algorithms | Anomaly Detection | CI/CD | Cloud platformClassPass membership | Culture Swap Days | Employee resource groups | Extra annual leave | Flexible work hoursSenior-level Full TimeSydney, Australia R17h ago
-
Senior Machine Learning Engineer AUD 144K-180KAWS | Algorithms | Anomaly Detection | Azure | CI/CDClassPass membership | Employee resource groups | Extra annual leave | Flexible work | Learning and developmentSenior-level Full TimeMelbourne, Australia R17h ago
-
Staff ML - GenAI Engineer, Voice & Speech INR 3584K-4680KAPI Development | AWS | Audio Alignment | Audio Processing | Cloud ComputingFlexible work hours | Overlap with India and US business hours | Remote work | Work from IndiaSenior-level Full TimeIndia R18h ago
-
Agile | Azure Data | Azure Data Factory | Azure Data Services | Azure SynapseContract opportunity | Fully remote CanadaMid-level Contract Full TimeCanada R19h ago
-
AWS | Alerting | Application Security | Authentication | AuthorizationAdditional Holiday Benefits | Continuous learning | Flexible remote first async working | Fully remote | High ownership roleSenior-level Full TimeIndia R19h ago
-
Auditing | Automation | Azure | Debugging | Enterprise IntegrationBring Your Own Device (BYOD) | Contractor engagement | Flexible work environment | Fully remote | No overtimeSenior-level Contract Full TimeBrazil R20h ago
-
Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure Data Lake Storage Gen2Contract opportunity | Direct influence on data strategy | Fully remote | High ownership | International collaborationSenior-level Contract Full TimeNetherlands R20h ago
-
Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure Data Lake Storage Gen2Contract opportunity | Fully remote | International collaborative environment | Ownership and influence | Part time to full time transition opportunitySenior-level Contract Full TimeIreland R20h ago
-
Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure Data Lake Storage Gen2Contract opportunity | Fully remote | High ownership | International collaboration | Part time discovery to full time implementation transitionSenior-level Contract Full TimeSwitzerland R20h ago
-
Azure | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake StorageContract opportunity | Fully remote | High ownership | Influence on enterprise data strategy | International collaborationSenior-level Contract Full TimeFrance R20h ago
-
Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure Data Lake Storage Gen2Contract opportunity | Enterprise data strategy influence | Fully remote | International collaboration | OwnershipSenior-level Contract Full TimeSpain R20h ago