AI Engineer, Quality
San Francisco, CA or Remote (USA) / Remote (US)
R
USD 170K-220K Entry-level Full Time
Tasks
- Build unified evaluation platform for AI agents
- Create automated model evaluation pipelines
- Create evaluation datasets from production traces
- Define evaluation standards and best practices
- Design guardrails and monitoring for quality regressions
- Develop evaluation harnesses and comparison frameworks
- Implement observability and tracing for agent execution
- Integrate and orchestrate LLMs tools and retrieval systems
Perks/Benefits
- 401k
- Flexible PTO
- Flexible work schedules
- Technology reimbursement
- Therapy sessions
- Wellness benefits
- Work from home reimbursement
Skills/Tech-stack
Embeddings | Evaluation Frameworks | LLM orchestration | Langfuse | Langgraph | Langsmith | Language Models | Large Language Models | Monitoring | Observability | PostgreSQL | Python | RAG | React | Retrieval-Augmented Generation | Tracing | TypeScript | Vector Databases
Education
N/A
Roles
Regions
Countries
States
Related jobs
-
Senior AI Engineer USD 190K-240KAgentic Systems | Anthropic API | Artificial Intelligence | Embeddings | Evaluation Frameworks401k plan | Flexible time off | Flexible work schedules | Health and wellness benefits | In-person offsitesSenior-level Full TimeSan Francisco, CA / Remote (US) R16h ago
-
Staff AI Engineer USD 240K-310KAgent Orchestration | Distributed Systems | Embedding pipelines | Embeddings | Guardrails401k plan | Flexible time off | Flexible work schedules | Health and wellness benefits | In-person offsitesSenior-level Full TimeSan Francisco, CA / Remote (US) R16h ago
-
AI Scientist - Machine Learning (US/KR) USD 136K-185KAblation Studies | Benchmarking | Data Analysis | Data Modeling | Exploratory Data AnalysisSenior-level Full TimePalo Alto, CA / Yeoksam, Seoul R1d ago
-
Senior Software Engineer I, Backend USD 130K-170KAgile | Alerting | Apache Airflow | Apache Spark | AutomationSenior-level Full TimeRemote, United States R1d ago
-
Senior Machine Learning Engineer USD 175K-230KAWS Bedrock | AWS SageMaker | Data Engineering | Experimentation | Language ModelsCase by case accommodation for medical beliefs | Case by case accommodation for religious beliefs | Remote work flexibilitySenior-level Full TimeUnited States R1d ago
-
Data Engineer USD 117K-195KAWS | Cloud Security | Cloud infrastructure | Data Governance | Data LakesHealth insurance | Holiday pay | Learning and development | Life insurance | Long-term disabilitySenior-level Full TimeUSA-Remote Work R1d ago
-
Senior Data & ML Engineer USD 151K-205KDBT | Data Governance | Data Observability | Data Quality | Data Security401k match | Dental insurance | Equity | Family planning resources | Flexible vacation policySenior-level Full TimeRemote - USA R2d ago
-
Senior Machine Learning Engineer USD 198K-287KArtificial Intelligence | Data Engineering | Fine Tuning | Foundation Models | GenAIOccasional travel for offsites | On-call rotation participation | Reasonable accommodation for applicantsSenior-level Full TimeRemote - US R2d ago
-
Senior AI Technologist USD 122K-180KAWS | Apache Airflow | Apache Spark | Azure | CI/CDFully remote option | Hybrid option | Onsite optionSenior-level Full TimeRaleigh, North Carolina, United States; Albuquerque, … R2d ago
-
Senior AI Systems Engineer USD 122K-188KAlerting | Bash | CI/CD | CMMC | Cause analysisFully remote option | Hybrid option | Onsite optionSenior-level Full TimeRaleigh, North Carolina, United States; Albuquerque, … R2d ago
-
Staff AI Engineer USD 200K-240KAWS | Agentic Workflows | ClickHouse | Distributed Systems | DockerParental leave | Tuition reimbursement | Unlimited PTOSenior-level Full TimeHybrid (NYC Metro) R2d ago
-
Senior AI Engineer USD 170K-205KAWS | Agent Orchestration | Agentic Workflows | CI/CD | ClickHouseHealth benefits | Parental leave | Stock options | Tuition reimbursement | Unlimited PTOSenior-level Full TimeHybrid (NYC Metro) R2d ago
-
AI Engineering Intern USD 54K-66KAPI Development | AWS | Azure | C++ | Code reviewAbility to travel for onsite orientation | Dedicated buddy | Intern workshops | Internship showcase | Remote internshipEntry-level InternshipUS-Remote R2d ago
-
AI Security Intern USD 50K-77KAWS Bedrock | Blue Teaming | Data leakage | Guardrails | Input validationDedicated buddy | Executive panel discussions | Intern events | Intern workshops | Internship showcase presentationEntry-level InternshipUS-Remote R2d ago
-
Staff Software Engineer, Data Infrastructure USD 221K-279KAWS | Apache Airflow | Apache Flink | Apache Iceberg | Apache KafkaAnnual refresh grants | Equity grants | In-person events | Remote work flexibilitySenior-level Full TimeUnited States - Remote R2d ago
-
Agile | Code review | Debugging | Java | JavaScriptMentorship | Paid apprenticeship | Remote workEntry-level Apprenticeship ContractRemote (United States) R2d ago
-
Agile | Data Analysis | Data Science | Machine Learning | PythonMentorship | Paid apprenticeship | Remote workEntry-level ContractRemote (United States) R2d ago
-
Staff Embedded Systems Engineer - US Based Remote USD 150K-175KAWS IoT | CAN | CI/CD | Data Telemetry | DockerSenior-level Full TimeUT; Remote (United States) R2d ago
-
Data Engineer - Healthcare (Remote) USD 185K-200KAWS | AWS Glue | AWS Glue Catalog | Amazon Athena | Amazon Bedrock401k matching | Dental insurance | Flexible time off | Flexible work hours | Medical insuranceMid-level Full TimeUnited States, United State, United States R2d ago
-
MLOps Engineer - Healthcare (Remote) USD 120K-140KAWS | Agile | Airflow | Alerting | Amazon CloudWatch401k matching | Dental insurance | Flexible time off | Medical insurance | Remote workMid-level Full TimeUnited States, UNITED STATES, United States R2d ago
-
Embedded Software Engineer USD 118K-198KBSP | Bare Metal | Bazel | Bus analysis | C#Community service time off | Dental insurance | Employee resource groups | Flexible family care days | Medical insuranceSenior-level Full TimeLong Beach, CA, United States R2d ago
-
Senior Embedded Software Engineer USD 135K-226KBSP | Bare Metal | Bazel | C# | C++Employee resource groups | Flexible family care days | Medical, dental & vision coverage | Paid Holidays | Paid parental leaveSenior-level Full TimeLong Beach, CA, United States R2d ago
-
AWS | Apache Spark | Azure | CI/CD | CloudFormationEquity | Remote workSenior-level Full TimeSan Francisco, CA R2d ago
-
Sr. Machine Learning Engineer USD 175K-230KAmazon Web Services | Apache Spark | C plus plus | Computer Vision | Data Preprocessing401k plan | Cell phone reimbursement | Dental insurance | Flexible paid time off | Health Savings Account contributionSenior-level Full TimeRemote - United States R3d ago
-
Senior-level Contract Full TimeSpringfield, IL - Remote R3d ago