Machine Learning Eval Engineer
Tasks
- Build customer specific benchmarks for real world workflows
- Build internal tooling for model visualization and evaluation analysis
- Build workflows to identify failure modes across datasets
- Design and build scalable evaluation systems for production LLM applications
- Design evaluation infrastructure for production AI systems
- Design statistical evaluation methodologies
- Develop benchmarks metrics and automated evaluation pipelines
- Own evaluation systems from design through production deployment
- Partner with ML engineers to prioritize model improvements using evaluation insights
- Prototype evaluation techniques using LLM as a judge
- Work with unstructured documents including PDFs spreadsheets and OCR outputs
Perks/Benefits
- Direct collaboration with founding teams
- Rapid Career Growth Opportunities
- Remote work
- Visa sponsorship available
Skills/Tech-stack
AWS S3 | Benchmarking | Debugging | Document AI | Document processing | Evaluation Pipeline | Experimentation | Flask | LLM Evaluation | LLM-as-a-Judge | Language Models | Large Language Models | Learning Metrics | Machine Learning | Machine learning metrics | OCR | Precisión | Prompt engineering | Python | Recall | Statistical Evaluation | TypeScript | Unstructured Data | Vision Language Models | Vision-language
Education
Bachelor of Engineering | Bachelor of Science | Master of Science | PhD
Regions
Countries
States
Related jobs
-
Featured Feat. Principal Knowledge & Data Architect USD 174K-284KAWS Neptune | Canonicalization | Chunking | Cypher | DBTBenefits including health and wellness programs | Health, wellness, and retirement plansSenior-level Full TimeHeadquarters - Chevy Chase, MD R12d ago
-
Featured Feat. Associate Director, Data Labs USD 167K-167KAWS | Cloud Computing | Compute Infrastructure | Data Analysis | LLM GovernanceConference speaking opportunities | Hybrid work schedule | Media appearancesSenior-level Full TimeWashington, District of Columbia, 20004, United … R25d ago
-
Artificial Intelligence | Automation | LLM integrations | Langchain | Language ModelsContractor | Fully remoteSenior-level ContractLos Angeles R9h ago
-
Lead Machine Learning Engineer USD 172K-276KAmazon Web Services | Azure | CI/CD | Cloud platform | DatabricksLearning and development programs | Mentorship and coachingSenior-level Full TimeChicago, Illinois, USA R11h ago
-
Solution Engineer USD 165K-216KAnalytics | Cloud Computing | Data Lake | Data Warehousing | Data fabricSenior-level Full TimeUS-FL-Remote R12h ago
-
Senior Data Engineer USD 126K-180KAmazon Kinesis | Apache Airflow | Apache Beam | Apache Flink | Apache Spark401k match | Bonus | Equity grant | Flexible time off | Health plansSenior-level Full TimeNew York, New York; Miami, Florida; … R12h ago
-
Senior Applied Scientist USD 170K-286KApache Spark | C++ | Computer Vision | Data Processing | Edge inferenceComprehensive health insurance | Parental leave | Professional development stipend | Remote work | Travel up to 5 percentSenior-level Full TimeRemote - US R13h ago
-
AI Integrations Staff Engineer USD 150K-230KAI Models | Agent Harness | Agent SDK | Back-end | Back-end development401k contribution | Company retreats | Dental insurance | Employee referral program | EquitySenior-level Full TimeRemote R15h ago
-
Applied AI Lead USD 120K-150KAgent workflows | Automated evals | Benchmarking | Continuous Monitoring | Evaluation Frameworks401k plan | Health insurance | Paid time off | Parental leave | Visa sponsorship not availableSenior-level Full TimeRemote/Hybrid R15h ago
-
Senior Data Engineer-JT0224 USD 115K-175K.Net Core | .Net Framework | AWS CloudFormation | Apache Airflow | Azure401k match | Career growth opportunities | Dental insurance | Employee resource groups | Health insuranceSenior-level Full TimeRemote, United States R16h ago
-
Senior Machine Learning Engineer USD 140K-224KAIOps | AWS | Apache Spark | Cloud platform | Data Analysis401k match | Dental insurance | Flexible paid time off | Life insurance | Medical insuranceSenior-level Full TimeRemote, United States R16h ago
-
WEM AQM Prompt Engineer USD 70K-195KAPI | Contact center | F1-score | Genesys | JSON401k employer matching | Basic life insurance | Company-Paid Holidays | Employee stock purchase plan | Health, dental, vision coverageMid-level Full TimeUnited States (Remote) R16h ago
-
Senior WEM AQM Prompt Engineer USD 88K-247KA/B | A/B Testing | API Integration | B testing | CalibrationInclusive environment | Remote work | Team-first cultureSenior-level Full TimeUnited States (Remote) R16h ago
-
Analytics Engineer USD 195K-264KAirflow | Cloud Data | Cloud Data Warehouse | DBT | Data ModelingCommuter stipend | Dental coverage | Flexible PTO | Lunch and dinner stipends | Medical coverageSenior-level Full TimeSan Francisco, CA & New York, … R16h ago
-
Staff Backend Engineer, AI Systems USD 181K-226KAPIs | Asynchronous processing | Azure OpenAI | Context retrieval | Distributed SystemsBenefits | Remote-first workSenior-level Full TimeUnited States Remote R17h ago
-
Artificial Intelligence | Assessment | Curriculum Development | Live coding | Machine LearningEntry-level Part TimeRemote, United States R17h ago
-
Cloud DevOps Engineer, Data & Analytics USD 195K-250KAWS | Azure | Bash | CI/CD | Disaster RecoveryHybrid work | Remote 2 days per week | Supportive collaborative environmentMid-level Full TimeAtlanta; Boston; Charlotte; Chicago; Dallas; Los … R17h ago
-
Staff Designated Support Engineer USD 122K-160KAI workflows | AWS | As-a-Service | Azure | Cloud DataSenior-level Full TimeUS-TX-Remote R18h ago
-
Senior, Machine Learning Engineer - 3D Perception USD 177K-212K3D Perception | Camera-based perception | Cloud processing | Computer Vision | Data Preparation401k employer match | Accidental death and dismemberment insurance | Flexible schedule | Holiday office closures | Life insuranceSenior-level Full TimeRemote, U.S, Ann Arbor, MI, Fort … R20h ago
-
Principal Data Engineer USD 135K-180KAWS | Airflow | Auditability | CI/CD | DBTCell phone allowance | Comprehensive health coverage | Equity grants | Home office setup allowance | Performance bonusesSenior-level Full TimeRemote US R20h ago
-
Asynchronous training | Containerization | Curriculum learning | Distributed Systems | Distributed TrainingSenior-level Full TimeSF Bay Area, CA, Remote, US, … R20h ago
-
Azure ML | Batch Processing | Cloud Machine Learning | Data Pipelines | Data Versioning401k employer match | Discretionary paid time off | Emotional & mental wellness support | Employee resource groups | Fitness programsMid-level Full TimeScottsdale, Arizona, United States R20h ago
-
Senior Software Engineer II (TASER Data Science) USD 148K-237KAzure ML | Batch Processing | Data Engineering | Data Pipelines | Databricks401k employer match | Emotional & mental wellness support | Employee resource groups | Fitness programs | Learning and development programsMid-level Full TimeWashington, United States R20h ago
-
Machine Learning Engineer / AI Model Developer (Mid to Senior)(Top Secret Clearance Required) (Hybrid) USD 77K-176KAPI Development | Agile | Backend Development | Cloud Computing | Data Preparation401k | Employee discounts | Employee referral rewards | Flexible work schedule | Health savings accountMid-level Full TimeQuantico, VA, US R22h ago
-
Machine Learning Engineer / AI Model Developer (Mid to Senior)(Top Secret Clearance Required) (Hybrid) USD 78K-176KAPIs | Agile methodologies | Backend Development | Cloud Computing | Data Preparation401k | Employee discount program | Employee referral rewards | Flexible spending account | Flexible work scheduleMid-level Full TimeFort Belvoir, VA, US R22h ago