Applied AI Lead - Evaluation & Measurement
Frederick, MD, US, 21703
R
USD 107K-157K (estimate) Senior-level Full Time
Tasks
- Build evaluation pipelines
- Connect measurements to ground truth systems
- Create measurement dashboards
- Define telemetry and trace logging
- Design evaluation plans
- Maintain record intake for AI ideas
- Produce monthly and quarterly results
- Set acceptance criteria and error budgets
Perks/Benefits
- 401k
- Community service hours
- Dental insurance
- Disability insurance
- Flexible work schedule
- Health Savings Account contribution
- Health and emergency leave
- Hybrid work
- Legal insurance
- Life insurance
- Medical insurance
- Paid parental leave
- Paid vacation
- Telemedicine
- Vision insurance
- Wellness programs
Skills/Tech-stack
Confidence Intervals | Cost accounting | Dashboards | Data Integrity | Data Pipelines | Error budget | Evaluation | Language Models | Large Language Models | Logging | Machine Learning | Model Validation | Python | Statistics | Telemetry
Education
Bachelor of Engineering | Bachelor of Science | Bachelor of Science in Computer Science | Bachelor of Science in Data Science | Bachelor of Science in Engineering
Roles
AI | AI Evaluation Lead | Engineer | Evaluation Lead | Lead | Learning Engineer | Machine Learning Engineer
Related jobs
-
C# | C++ | Computer Graphics | Computer Vision | Data PipelinesEquity compensation | Health benefits | Onsite work schedule Tuesday Wednesday ThursdaySenior-level Full TimeSan Mateo, CA, United States R6h ago
-
Senior Director of Engineering, Generative AI USD 525K-573K3D Scene | 3D Scene Understanding | Agentic Systems | Artificial Intelligence | BenchmarkingEquity compensation | Health benefits | Onsite days Tuesday Wednesday ThursdaySenior-level Full TimeSan Mateo, CA, United States R7h ago
-
Senior Machine Learning Engineer, Content Platform USD 141K-360KApache Spark | Cloud Computing | Generative AI | Language Models | Large Language ModelsDisability benefits | Equity awards | Financial wellness support | Health insurance | Life insuranceSenior-level Full TimeSan Jose, California R9h ago
-
Staff Machine Learning Engineer, Ad Serving USD 195K-510KA/B | A/B Testing | B testing | Continuous Improvement | Feature EngineeringCommuter benefits | Disability benefits | Financial wellness support | Health insurance | Life insuranceSenior-level Full TimeNew York, New York R9h ago
-
Staff Machine Learning Engineer, Ad Serving USD 195K-435KDistributed Systems | Experimentation | Feature Engineering | Java | Machine LearningDisability benefits | Equity awards | Health insurance | Life insurance | Paid time offSenior-level Full TimeBoston, Massachusetts R9h ago
-
AI Agents | AI monitoring | API Integration | Amazon Web Services | Anthropic401k retirement plan | Medical, dental, and vision benefits | Paid Holidays | Paid time off | Variable pay/incentivesSenior-level Full TimeUnited States R10h ago
-
Senior Manager, Commercial Analytics (Immunology - HLH) USD 140K-175KClaims data | Competitive Intelligence | Dashboard Design | Data Mart | Data VisualizationCompetitive 401k match | Conference attendance | Generous time-off policy | Hybrid remote work | Medical, dental, and vision coverageSenior-level Full TimeWaltham, MA, United States R10h ago
-
Machine Learning Engineer, Ads USD 165K-230KA/B | A/B Testing | B testing | Bidding | Conversion PredictionDirect impact | Hybrid work model | International team collaboration | Professional growth opportunities | Remote work flexibilityMid-level Full TimeSan Francisco Bay Area, USA R10h ago
-
Team Lead, AI Engineering USD 157K-254KAI Search | AKS | Agent workflows | Async Programming | AuthenticationRemote work hybrid flexibilitySenior-level Full TimeUSA - Atlanta, GA; USA - … R13h ago
-
Lead Data Engineer USD 164K-200KAPIs | AWS | Agentic AI | Azure | Batch dataDisability and life insurance | Employee assistance program | Fertility and family forming benefits | Flexible spending account | Healthcare conciergeSenior-level Full TimeHeadquarters, United States R13h ago
-
Sr. Analytics Engineer USD 160K-185KAirflow | CI/CD | DBT | Dagster | Data ModelingCross-functional collaboration | High autonomy | US Pacific overlapSenior-level Full TimeRemote - US R14h ago
-
AWS | Cloud infrastructure | Compute | Data Curation | Distributed Systems401k match | Daily lunches and snacks | Health insurance covered | Hybrid work schedule | Learning and development stipendSenior-level Full TimeRedwood City R15h ago
-
Sr. Forward Deployed Engineer (FDE) - Public Sector USD 182K-250KAI APIs | AWS | Apache Spark | CI/CD | Cloud platformCustomer-facing role | Hybrid work | Secret clearance required | Travel 20 to 25 percentSenior-level Full TimeAlabama; Arkansas; Florida; New York; Tennessee R15h ago
-
Staff Robotics Engineer USD 190K-235KAI Agents | AI Libraries | C++ | Critical Systems | FDIRComprehensive employee benefits | Equity | On call rotation for continuous campaignsSenior-level Full TimeRichmond, CA R16h ago
-
AI Researcher — Intern USD 54K-70KAgentic Systems | Benchmarking | Calibration | Data Curation | Data evaluationHybrid work environment | Opportunity for publication | Research mentorshipEntry-level InternshipNew York City, NY (Hybrid) R16h ago
-
Senior Data Engineer USD 130K-170KAmazon Kinesis | Amazon Redshift | Apache Airflow | BigQuery | DBT401k match | Education stipend | Flexible time off | Health insurance | IKEA discountsSenior-level Full TimeSan Francisco, California, United States R16h ago
-
Sr. Staff Machine Learning Systems Engineer USD 240K-265KAdversarial Testing | Calibration | Data Engineering | Data Ingestion | Data Pipelines401k employer matching | Company holidays | Comprehensive health benefits | Employee stock purchase program | Offsite team retreatsSenior-level Full TimeUS Remote R16h ago
-
Staff Backend AI Engineer, Remote USD 164K-219KAWS | AWS Cloud | AWS Cloud Development Kit | AWS Lambda | AWS cloud development401k matching | Dental insurance | Flexible time off | Medical insurance | Paid HolidaysSenior-level Full TimeUnited States, UNITED STATES, United States R17h ago
-
Junior AI Engineer (Open to remote) USD 110K-135KAPIs | Agent SDK | Agentic coding | Arize | Google ADK401k | Commuter benefits | Dental insurance | Educational assistance | Medical insuranceExecutive-level Full TimeNew York, United States R17h ago
-
Staff AI Architect, Remote USD 160K-236KAPI Gateway | AWS CDK | AWS ECS | AWS Secrets | AWS Secrets ManagerDental insurance | Flexible time off | Hybrid work | Matching 401k | Medical insuranceSenior-level Full TimeUnited States, UNITED STATES, United States R17h ago
-
Senior-level Full TimeCalifornia; Remote - California; San Francisco, … R17h ago
-
Principal Machine Learning Engineer, Geometric Vision USD 407K-460K3D Computer Vision | 3D representations | Auto-labeling | C++ | CalibrationSenior-level Full TimeSunnyvale R18h ago
-
Media Software Engineer, Speech (Senior-Staff Levels) USD 180K-270KAndroid | Asynchronous I/O | Audio streaming | C# | C++401k retirement savings plan | Company holidays | Complimentary lunch and snacks | Fertility support | Lifestyle spending accountSenior-level Full TimeSunnyvale R18h ago
-
AI/ML Engineer USD 100K-150KAPI Design | Asynchronous programming | Claude Code | Cloud platform | Cursor401k | Employee assistance program | Health spending account | Life insurance | Long-term disabilityMid-level ContractDearborn, Michigan, USA R18h ago
-
Machine Learning Engineer, Detection and Tracking USD 160K-210KByteTrack | DETR | Data Augmentation | DeepSORT | Faster R-CNNFlexible paid time off | Hybrid work | Medical insurance | Paid Holidays | Remote workMid-level Full TimeWashington, DC R18h ago