Senior Machine Learning Engineer, AI Evaluation
Tasks
- Apply inter rater reliability and model as judge methodologies
- Build data repositories for experiment results and metadata
- Build provider agnostic evaluation orchestration layer
- Design scalable AI evaluation infrastructure
- Develop reproducible experimentation with run logging and experiment tracking
- Ensure auditable documentation and evaluation transparency
- Ensure evaluation systems comply with data security privacy governance
- Implement AI evaluation scoring frameworks
- Implement access controls and protect sensitive data
- Implement monitoring reporting and visualization
- Maintain model version pinning and drift detection
- Recommend evaluation methodologies and emerging tool adoption
- Support design when ground truth is unavailable
- Translate HR standards into measurable evaluation specifications
Perks/Benefits
- N/A
Skills/Tech-stack
AI Evaluation | Artificial Intelligence | Audit Logging | Benchmarking | BigQuery | Cloud platform | Data Governance | Data Pipelines | Drift Detection | Experiment tracking | Google Cloud | Google Cloud Platform | IAM | Inter-rater reliability | Language Models | Large Language Models | Logging | Looker | Looker Studio | Machine Learning | Model as judge | Privacy | Prompt engineering | Provider Agnostic | Reproducibility | Rubric Scoring | Security | Version control | Vertex AI
Education
Regions
Countries
States
Cities
Related jobs
-
AC DC Sizing | AC/DC | Arc Flash | Artificial Intelligence | AutomationMultiple project extensions | Paid on time | Remote work optionSenior-level Full TimeJuno Beach, FL R10h ago
-
AI Agents | API Development | Azure | CI/CD | LLMOpsCollaborative cross-functional team | Fully remoteMid-level Full TimeCanada, Ontario; Toronto, None; Toronto, Ontario; … R10h ago
-
API Integration | AWS Bedrock | Auditing | Authentication | AuthorizationMentoring | MentorshipMid-level Full TimeMenlo Park, CA, United States11h ago
-
Lead Palantir AI and Data Science Engineer II USD 159K-265KAI workflows | Analytics | Cloud Computing | Dashboards | Data GovernanceProfessional development | Travel opportunitiesSenior-level Full TimeRosslyn, Virginia, United States14h ago
-
Analytics and Insights Engineer III USD 131K-218KApache Spark | Data Governance | Data Modeling | Data Pipelines | Data QualitySenior-level Full TimeRichmond, Virginia, United States14h ago
-
3D Rendering | A/B | A/B Testing | Agentic AI | AlgorithmsSenior-level Full TimeSan Jose, California, United States14h ago
-
C++ | Computer Vision | Data Processing | Deep learning | Domain AdaptationEntry-level Full TimeSan Jose, California, United States14h ago
-
Staff Software Engineer, AI and Infrastructure USD 207K-300KC++ | Cluster management | Distributed Systems | Kernel | LinuxSenior-level Full TimeSunnyvale, CA, USA15h ago
-
Staff Software Engineer, AI/ML, Google Cloud Security USD 207K-300KAgentic AI | Artificial Intelligence | Cloud Security | Cloud platform | Data ProcessingSenior-level Full TimeReston, VA, USA; New York, NY, …15h ago
-
Senior Software Engineer, Google Cloud Dataproc USD 174K-252KApache Hive | Apache Spark | C++ | Data Warehouse | Data Warehouse InternalsSenior-level Full TimeSunnyvale, CA, USA15h ago
-
Agentic Orchestration | C++ | Data Onboarding | Data Pipeline Engineering | Data pipelineSenior-level Full TimeSan Jose, CA, USA15h ago
-
Senior Software Engineer, AI/ML, Core USD 174K-252KC++ | Data Processing | Data Structures | Data Structures and Algorithms | DebuggingSenior-level Full TimeMountain View, CA, USA15h ago
-
Staff Software Engineer, Search Local Ads Data Platform USD 207K-300KC plus plus | CSS | Data Storage | Data Structures | Data Structures and AlgorithmsSenior-level Full TimeMountain View, CA, USA15h ago
-
Principal Hardware Robotics Engineer, Data Centers USD 275K-382KActuator control | Embedded Systems | FPGA | Haptic feedback | Human-in-the-loopSenior-level Full TimeSunnyvale, CA, USA15h ago
-
Analytics Data Engineer USD 94K-130KAzure Data | Azure Data Factory | CI/CD | Data Analysis | Data FactoryRemote workMid-level Full TimeFlagstaff, AZ, US21h ago
-
Benchmarking | C plus plus | CUDA | Computational Engineering | Distributed Computing401k match | Generous paid time off | Medical, dental & vision coverage | Stock options | WFH Hardware ReimbursementSenior-level Full TimeSunnyvale, CA R22h ago
-
Associate, Client Intelligence (Data Scientist) USD 98K-132KAWS | Apache Spark | Azure | Big Data | Cloud platformEmployee resource groups | Matching gift program | Professional development | Sustainability and Climate Change Initiatives | Volunteerism programMid-level Full TimeNew York, NY, US, 100011d ago
-
Apple Neural Engine | C++ | CPU | Compilers | Computer ArchitectureMid-level Full TimeSeattle1d ago
-
Staff ML Engineer, Foundation Models USD 251K-310KBayesian Inference | Foundation Models | Generative Modeling | Hierarchical learning | Language ModelsBonus | Equity incentiveSenior-level Full TimeMountain View, CA, USA R1d ago
-
Apple Neural Engine | C plus plus | CPU performance | Compilers | Computer ArchitectureMid-level Full TimeCupertino1d ago
-
Senior Machine Learning Engineer USD 160K-200KAWS | CI/CD | Computer Vision | Data Processing | DockerCommuter benefits | Dental insurance | Employee assistance program | Equity | Health insuranceSenior-level Full TimeArlington, VA R1d ago
-
Senior Computer Vision & Machine Learning Engineer USD 140K-206KAblation Studies | CI/CD | Data Pipelines | Docker | Error AnalysisAutonomy | Client project impact | Support for professional growthSenior-level Full TimeRemote, US R1d ago
-
Computer Vision & Machine Learning Engineer USD 150K-200KAblation Studies | CI/CD | Computer Vision | Data Pipelines | DockerEntry-level Full TimeRemote, US R1d ago
-
C# | C++ | Fine Tuning | Foundation | ICUMid-level Full TimeCupertino1d ago
-
Staff Tech Lead, Machine Learning Engineer, Perception USD 251K-310KC plus plus | Computer Vision | Data pipeline | JAX | Machine LearningAnnual bonus | Company benefits program | Equity incentive planSenior-level Full TimeMountain View, California, United States; San …1d ago