AI Engineer, Quality
San Francisco, CA or Remote (USA)
R
USD 170K-220K Senior-level Full Time
Tasks
- Advocate for evaluation-driven development
- Build automated model evaluation pipelines
- Create evaluation datasets with SMEs
- Define evaluation standards and best practices
- Design and build unified evaluation platform
- Design evaluation harnesses and comparison frameworks
- Design prompts and orchestration systems
- Develop observability systems for production
- Implement quality guardrails and monitoring
- Integrate LLMs, tools, retrieval systems
- Own evaluation infrastructure stack integration
- Own large product areas related to quality and evaluation
- Partner with product and ML engineers on evaluation requirements
- Translate customer problems into agent behaviors
- Use AI for design, testing, and iteration
Perks/Benefits
- N/A
Skills/Tech-stack
AI-native | AI-native engineering | Agent Orchestration | Automation | Data-Driven Decision Making | Data-driven | Decision Making | Embedding Models | Evaluation Frameworks | LLMs | Machine Learning | Monitoring Systems | Observability | Postgres | Prompt engineering | Python | RAG Architectures | React | Tracing infrastructure | TypeScript | Vector Databases
Education
N/A
Roles
Regions
Countries
States
Related jobs
-
Senior AI and LLM Engineer USD 184K-356KData-driven | Data-driven metrics | Deep learning | Generative AI | Language ModelsEquity | Hybrid workSenior-level Full TimeUS, CA, Remote, United States R17h ago
-
AI Engineer USD 53K-119KAPI Design | Cost Optimization | Embeddings | Evaluation | JSONDental insurance | Gym stipend | Health insurance | Medical membership | Offsite retreatsSenior-level Full TimeRemote, US R20h ago
-
Senior Forward Deployed AI Engineer (Remote Eligible) USD 227K-245KAWS Bedrock | Agent Framework | Agent Orchestration | Cloud infrastructure | Databricks401k match | Dental insurance | Flexible time off | Health insurance | Life insuranceSenior-level Full Time-REMOTE, USA- R1d ago
-
AI Analytics and Visualization Engineer USD 152K-287KAPI Development | AWS | Azure | Backend Services | Cloud platformSenior-level Full TimeUS, CA, Remote, United States R1d ago
-
Bayesian estimation | Cloud deployment | Data Cleansing | Data Wrangling | Decision and ControlContract-to-hire option | Flexible schedule | Remote workSenior-level Contract Full TimePalo Alto, CA, United States R1d ago
-
Data and AI Engineer USD 120K-165KAWS | Agile | Azure | Big Data | Data ArchitecturePublic trust clearance required | Remote workMid-level Full TimeWork from home, VA, United States R2d ago
-
Sr. Data and AI Engineer USD 180K-200KAgile | Amazon Web Services | Azure | Big Data | Data ArchitecturePublic trust clearance support | Remote workSenior-level Full TimeWork from home, VA, United States R2d ago
-
Agile | Artificial Intelligence | Code review | Debugging | DocumentationMentorship | Remote work | Tuition coverageEntry-level Apprenticeship ContractRemote (United States) R2d ago
-
Ai / Ml Sme Iii USD 100K-135KArtificial Intelligence | Automation | Communications Protocols | Cybersecurity | Information Systems401k | Dental insurance | Disability insurance | Health insurance | Life insuranceSenior-level Full TimeRemote, United States R2d ago
-
Senior Forward Deployed AI Engineer USD 106K-180KAWS | Automation | CI/CD | Distributed Systems | EmbeddingsBenefits | Bonus eligibility | Remote work optionSenior-level Full TimeUnited States - Remote R2d ago
-
Staff Software Engineer, AI/ML USD 136K-265KAutogen | Cloud Platforms | CrewAI | Embeddings | Generative AIContinuous learning | Remote work | Vacation packageSenior-level Full TimeRemote - USA, United States R2d ago
-
Sr. Staff Embedded AI Engineer USD 145K-185KBare Metal | C# | C++ | CMSIS NN | Code generationEmployee resource groups | Flexible work environment | Hybrid work model | Remote work optionSenior-level Full TimeColumbia, MARYLAND, United States R2d ago
-
AI Engineer (Hybrid) USD 130K-270KAmazon Web Services | Cloud Computing | Data Analysis | Data Modeling | Data Quality401k contribution | AD&D insurance | Health Savings Account contribution | Hybrid work | Life insuranceSenior-level Full TimeAnnapolis Junction, MD R3d ago
-
Senior AI Engineer USD 150K-210KAPI Integration | Cloud Native | Cloud Native Services | Container Orchestration | EmbeddingsBenefits | Equity | High-trust work environment | MentorshipSenior-level Full TimeSF Bay Area (Hybrid) R3d ago
-
AML | APIs | AWS | Access Management | AzureSenior-level Full TimeNew Jersey Office - 210 Hudson … R3d ago
-
Staff AI Engineer USD 175K-250KAgent systems | Backtesting | Data Pipelines | Distributed Systems | Fine TuningBonus eligibility | Equity | Performance incentives | Remote work | Token participationSenior-level Full TimeNew York, United States - Remote R3d ago
-
Applied & Agentic AI Engineer USD 105K-115KAPI Integration | Access Control | Agent systems | Audit Logging | Azure OpenAIGreat place to work | Work-life balanceSenior-level Full TimeTelecommuter TN, United States R3d ago
-
HCL Informix | HCL Vector Blade | LLM | MCP | PythonDedicated buddy | Executive Networking | Intern events | Internship showcase | Onsite orientation travelEntry-level InternshipUS-Remote R3d ago
-
Senior-level Full TimeRemote, XX, United States R3d ago
-
AWS | Azure | C# | C++ | Cloud PlatformsHousing assistance | Mentorship | Opportunity for return offers | Professional development | Real-world project ownershipEntry-level Full TimeEvendale, United States R4d ago
-
APIs | Authentication | Authorization | CI/CD | Code reviewSenior-level Full Time3401 WEST END AVE (LOC00541), United … R4d ago
-
4109- Applied AI & Analytics Consultant USD 155K-210KAPI | AWS | Azure | Classification | DBTBasic life insurance | Dental insurance | Discounted Legal Aid | Long-term disability | Medical insuranceSenior-level Full TimeUnited States - Remote R4d ago
-
AI Engineer USD 85K-95KAWS | Azure | Data Analysis | Data Governance | Data Security401k retirement plan | Commuter benefits | Dental insurance | Employee assistance program | Family support programSenior-level Full TimeUS - Morrisville, NC - 3900 … R4d ago
-
Associate Director, AI Engineer USD 94K-152KAPI Architecture | Autogen | Azure | CI/CD | Container Orchestration401k | Flexible paid time off | Life insurance | Long-term disability | Medical/Dental/Vision insuranceMid-level Full TimeUSA - Remote - Maryland, United … R4d ago
-
Senior AI Engineer USD 190K-240KAgentic Systems | Anthropic API | Artificial Intelligence | Embeddings | Evaluation Frameworks401k plan | Flexible time off | Flexible work schedules | Health and wellness benefits | In-person offsitesSenior-level Full TimeSan Francisco, CA / Remote (US) R4d ago