Senior AI Quality Engineer (LLM Evaluation & Automation) 1754
Tasks
- Build eval harness
- Create regression packs
- Define scorecard metrics
- Develop adversarial testing plan
- Develop drift testing plan
- Fail builds on quality regression
- Generate candidate eval cases with AI
- Integrate evals into CI
- Maintain exception tasks
- Maintain golden tasks
- Partner with product and tech lead on thresholds
- Set release gate thresholds
Perks/Benefits
Skills/Tech-stack
Automation | Benchmarking | CI/CD | Case design | LLM Evaluation | Language Models | Large Language Models | Machine Learning | Probabilistic Modeling | Regression testing | Scripting | Test Case | Test Case Design
Education
N/A
Related jobs
-
AWS | Apache Spark | Automation | Azure | CI/CDLearning and development opportunities | Mentorship | Remote work flexibility | Supportive corporate cultureSenior-level ContractAmsterdam, NH, Netherlands R1d ago
-
LLM Engineer bei der JUST ADD AI (all genders) EUR 53K-59KAgent workflows | Fine Tuning | Hosting | LLM security | Language ModelsAnnual offsites | Businessbike | E-learning support | Extra holiday | Fitness benefitsMid-level Full TimeBremen // Hamburg // Hybrid R1d ago
-
Senior Machine Learning Engineer USD 145K-230KChromaDB | Data Pipelines | Deep learning | Faiss | Hugging FaceCompany sponsored training | Home internet support | Hybrid work model | Medical insurance | Mobile allowanceSenior-level Full TimeDhaka, Bangladesh R1d ago
-
Data Engineer to Financial Crime Prevention | SEB, Solna SEK 775K-1045KApache Beam | Automated testing | CI/CD | Cloud Computing | Data ModelingContinuous improvement culture | Hybrid work | Learning and coaching | Work-life balanceSenior-level Full TimeSolna R1d ago
-
Elasticsearch | Experimentation | Information Retrieval | Language Processing | Learning-to-RankEquity packages | Flexible leave options | Inclusive parental leave | Remote work within Australia | Wellbeing allowanceSenior-level Full TimeSydney, NSW, Australia R1d ago
-
Elasticsearch | Experimentation | Information Retrieval | Learning-to-Rank | Machine LearningEquity package | Flexible leave | Parental leave | Remote work within Australia | Wellbeing allowanceSenior-level Full TimeMelbourne, Australia R1d ago
-
Agent systems | Backend Development | Bias Testing | Cost Optimization | Data Science401k | Flexible paid time off | Healthcare | Home office stipend | Inclusive cultureSenior-level Full TimeCanada R1d ago
-
C# | C++ | Distributed Systems | Experimentation | GoAccident insurance | Commuter benefits | Dental insurance | Disability insurance | Financial wellness supportSenior-level Full TimeCambridge, United Kingdom R1d ago
-
Senior Machine Learning Engineer, Search USD 361K-510KA/B | A/B Testing | Apache Airflow | Apache Spark | B testingCommuter benefits | Disability benefits | Equity awards | Health insurance | Life insuranceSenior-level Full TimeSan Jose, California R1d ago
-
Access Controls | Airflow | Apache Spark | Asset bundles | CI/CDCertification incentives | Childcare allowance | Dental insurance | Educational discounts | Fitness and wellness benefitSenior-level Full TimeBrazil R1d ago
-
Artificial Intelligence | Automation | Bash | Block Storage | CSICareer growth potential | Equal opportunity | Inclusive environment | International and diverse working environment | Professional developmentSenior-level Full TimeGermany R1d ago
-
Kaipūhanga Raraunga Mātāmua Senior Data Engineer NZD 117K-155KAutomated data quality | Azure | Azure Data | Azure Data Factory | Azure Data LakeBuy additional leave | Career development | Enhanced employer KiwiSaver contribution | Enhanced sick leave | Hybrid workingSenior-level Full TimeWellington, Wellington Region, New Zealand R1d ago
-
AWS Glue | AWS Lake | AWS Lake Formation | Amazon Athena | Amazon BedrockChildcare assistance | Continuous learning opportunities | Discounts through partner programs | Extended parental leave | Fitness and wellness programsMid-level Full TimeBrazil R1d ago
-
Senior Embedded Software Engineer USD 120K-160KAADL | ADA | ARINC 653 | Agile | Atlassian Bitbucket401k match | Career development | Employee assistance program | Low-cost medical dental vision | Paid HolidaysSenior-level Full TimeDenver, CO R1d ago
-
Senior Applied Scientist COP 88000K-94620KAWS Bedrock | AWS EKS | AWS Lambda | AWS OpenSearch | AWS TextractFuneral assistance | Home office stipend | Internet allowance | Life insurance | Paid time offSenior-level Full TimeBogotá, Colombia R1d ago
-
Senior Applied Scientist COP 88000K-94620KAWS | AWS AgentCore | AWS Bedrock | AWS Lambda | AWS TextractFuneral assistance | Home office stipend | Internet allowance | Life insurance | MentorshipSenior-level Full TimeBogotá, Colombia R1d ago
-
Azure Cloud Engineer, Data Mesh and AI Platforms team PLN 237K-360KAI Foundry | AKS | Application Insights | Azure | Azure AIHybrid work up to 40 percent remote possibleSenior-level Full TimeGdańsk, PL, 80-309 R1d ago
-
Senior AI Lead AUD 142K-156KAI Agents | Artificial Intelligence | Conversational AI | Data Analysis | Evaluation FrameworksCounseling access | Employee discounts | Employee micro credentials | Employee networks | Flexible working arrangementsSenior-level Full TimeSydney, Australia R1d ago
-
Lead AI Engineer EUR 64K-78K.NET | AIOps | Agentic AI | BMC CMDB | DockerBe Well programs | Certification support | Coaching | Continuous feedback | Flexible work environmentSenior-level Full TimeMadrid HQ (KES51610), Spain R1d ago
-
AI Engineer with Java USD 119K-209KAgentic AI | App Service | Azure App | Azure App Service | Azure CognitiveRemote workEntry-level Full TimeRemote, KA, IN R1d ago
-
AWS | AWS Glue | Agile | Amazon S3 | CI/CDSenior-level Full TimeHyderabad, Telangana, India - Remote R1d ago
-
AI Agents | API Integration | Artificial Intelligence | Automation | ChatbotsFlexible schedule | Inclusive international environment | No time tracking | Professional development | Remote workSenior-level Contract Full TimeSouth Africa - Remote R1d ago
-
AI Agents | API Integration | Automation | Chatbots | Conversational AIFlexible workload balance | Health policy | Inclusive international environment | No time tracking requirements | Remote work optionMid-level Contract Full TimeKyiv, Kyiv city, Ukraine - Remote R1d ago
-
Senior AI and Machine Learning Engineer INR 3000K-4680KConvolutional Neural Networks | Data Mining | Data Preprocessing | Data Visualization | Decision TreesCareer Development Programs | Flexible work arrangements | Health and wellbeing benefits | Hybrid work schedule | Inclusion initiativesSenior-level Full TimeBengaluru, Karnātaka, India R1d ago
-
Senior AI Machine Learning Engineer 100% (f/m/d) CHF 126K-150KAPIs | Apache Spark | Azure | CI/CD | Data PipelinesFlexible working arrangements | Hybrid working | Job-sharing | Part-time optionsSenior-level Full TimeGlattbrugg R1d ago