AI Engineer, Quality
San Francisco, CA or Remote (USA)
R
USD 170K-220K Senior-level Full Time
Tasks
- Advocate for evaluation-driven development
- Build automated model evaluation pipelines
- Create evaluation datasets with SMEs
- Define evaluation standards and best practices
- Design and build unified evaluation platform
- Design evaluation harnesses and comparison frameworks
- Design prompts and orchestration systems
- Develop observability systems for production
- Implement quality guardrails and monitoring
- Integrate LLMs, tools, retrieval systems
- Own evaluation infrastructure stack integration
- Own large product areas related to quality and evaluation
- Partner with product and ML engineers on evaluation requirements
- Translate customer problems into agent behaviors
- Use AI for design, testing, and iteration
Perks/Benefits
- N/A
Skills/Tech-stack
AI-native | AI-native engineering | Agent Orchestration | Automation | Data-Driven Decision Making | Data-driven | Decision Making | Embedding Models | Evaluation Frameworks | LLMs | Machine Learning | Monitoring Systems | Observability | Postgres | Prompt engineering | Python | RAG Architectures | React | Tracing infrastructure | TypeScript | Vector Databases
Education
N/A
Roles
Regions
Countries
States
Related jobs
-
Artificial Intelligence | Machine Learning | Marketing | Product Development | Product ManagementCoaching | Community access | Relocation support | Startup hiring support | Weekly founder sparringMid-level ContractAustin, United States R1d ago
-
Senior AI Engineer USD 170K-200KAgent systems | Agentic AI | Anthropic API | Automated evals | Backend architecture401k | Company-provided equipment | Comprehensive medical, dental and vision coverage | Disability insurance | Flexible vacation policySenior-level Full TimeRemote (United States) R1d ago
-
Senior AI Solutions Builder, People USD 134K-192KAgent Development | Artificial Intelligence | Automation | Coding | Data GovernanceSenior-level Full TimeRemote, United States R1d ago
-
Lead AI Engineer USD 200K-215KA/B | A/B Testing | AWS Bedrock | Agentic LLM | Agentic LLM systemsDental insurance | Employee discounts | Employee equity | Health insurance | Pet insuranceSenior-level Full TimeRemote - United States R1d ago
-
Agentic AI Engineering Intern USD 48K-86KAgent systems | Autogen | Automated testing | CI/CD | CrewAIEntry-level Full Time Internship6314 Remote/Teleworker US, United States R2d ago
-
AI Engineer USD 91K-175KAI Studio | AWS | Apache Spark | Azure | Azure AIHealth benefits | Incentive programs | Professional development | Remote work | Retirement benefitsSenior-level Full TimeWork at Home - Cincinnati - … R2d ago
-
Gen AI Engineer USD 132K-230KAWS ECS | AWS EKS | AWS Lambda | AWS Step Functions | Adversarial promptsHybrid work modelSenior-level Full TimeNew York/Dallas/Remote, NY, US R2d ago
-
AI Engineer USD 69K-125KAPI Integration | Autogen | Bias Mitigation | CUDA | ContainerizationSecret clearanceMid-level Full Time6314 Remote/Teleworker US, United States R2d ago
-
3803-AI Forward Deployed Engineer USD 117K-168KAI Agents | API Integration | AWS | Autogen | AzureDental insurance | Discounted Legal Aid | Health insurance | Life insurance | Long-term disabilityMid-level Full TimeUnited States - Remote R2d ago
-
AI Engineer USD 150K-215KAgent Orchestration | Agent systems | AgentOps | Amazon Bedrock | Azure OpenAIMid-level Full TimeRemote United States R2d ago
-
AI Engineer, Go-To-Market USD 155K-240KAI Agents | Artificial Intelligence | Attribution Modeling | Automation | CRMMid-level Full TimeUS - Remote R2d ago
-
AI & ML Engineering Coaches (Contract) USD 140K-242KArtificial Intelligence | Computer Vision | Language Processing | Machine Learning | Machine Learning InfrastructureFlexible schedule | Virtual sessionsSenior-level ContractRemote - Utah R2d ago
-
Senior Generative AI Engineer USD 125K-156KAI Governance | AWS | Bedrock | Data Processing | ECS401k plan | Dental insurance | Disability insurance | Employee referral program | Fertility benefitsSenior-level Full TimeUS Remote R3d ago
-
Staff Applied AI Engineer - Pre-Sales USD 190K-330KAgent architectures | Data Processing | Fine Tuning | GenAI | LLMCareer growth opportunities | Equity | Remote work optionsSenior-level Full TimeNew York City, NY (Hybrid); Redwood … R3d ago
-
AI Engineer III USD 118K-230KA2A | AI Foundry | Agent communication | Agent communication protocols | Agent evaluation401k employer match | Employee assistance programs | Employee recognition program | Health and wellness resources | Life and disability insuranceSenior-level Full TimeWA Western Telecommuter, United States R3d ago
-
Senior-level Full TimeVirginia - Remote Office, United States R3d ago
-
Senior Engineer 2: Inference Data Plane USD 167K-209KAI/ML | Cloud infrastructure | Distributed Systems | GPU Optimization | GRPCEmployee assistance program | Equity compensation | Flexible time off | Health benefits | Reimbursement for trainingSenior-level Full TimeDenver R3d ago
-
Sr. AI Engineer (24 months fixed-term) USD 137K-232KBias Mitigation | Communication | Data Analysis | Education Technology | Generative AIFlexible schedule | Health insurance | Inclusive culture | Learning and development opportunities | Paid time offSenior-level TemporaryMountain View, CA / Remote (Continental … R4d ago
-
PhD Intern - AI/ML/NLP Engineer USD 70K-86KAI platforms | Bayesian Methods | C# | CI/CD | Cloud ComputingDental insurance | Health insurance | Paid time off | Retirement plan | Tuition reimbursementEntry-level Full Time InternshipCalifornia - Remote Office, United States R4d ago
-
AI & Data Engineering Intern USD 56K-60KAI tools | AWS | Big Data | Data Analysis | Data PipelinesFlexible schedule | Remote workEntry-level Full Time InternshipHome based-Pennsylvania, United States R4d ago
-
Senior AI Engineer USD 166KAPIs | JSON | JavaScript | LLMs | MoveworksDental insurance | Flexible benefits | Health insurance | Paid time off | Retirement planSenior-level Full TimeUSA-NY, United States R4d ago
-
AWS | Azure | C# | C++ | Computer VisionHealth benefits | Paid time off | Parental leave | Professional development | Remote workExecutive-level Full TimeCambridge, United States R4d ago
-
Senior AI Engineer USD 166KA/B | A/B Testing | Automation Hub | B testing | Databricks401k match | Health savings account | Health, dental, and vision insurance | Life insurance | Long-term disability leaveSenior-level Full TimeUSA-NY, United States R4d ago
-
A/B | A/B Testing | AI Search | APIs | Alerting401k | ESPP | Health care benefits | Paid time off | Success sharing bonusSenior-level Full TimeGEORGIA - VIRTUAL - GA01, United … R4d ago
-
Agile Development | Data Science | Java | JavaScript | Machine LearningMentorship | Paid apprenticeship | Remote work | Tuition coverageEntry-level ContractRemote (United States) R4d ago