AI Engineer – Algorithm Evaluation & Agentic Systems
Tasks
- Build and evaluate agentic systems
- Build automated regression testing pipelines
- Conduct human in the loop evaluation
- Define evaluation metrics and KPIs
- Design evaluation frameworks and test infrastructure
- Develop multi step reasoning planning and memory workflows
- Evaluate image and video understanding models
- Implement LLM VLM judge evaluation
- Lead model benchmarking and integration
- Perform edge case failure analysis
- Run inference and extract embeddings
- Translate evaluation findings into improvement recommendations
Perks/Benefits
- N/A
Skills/Tech-stack
Agent systems | Automated Regression | Automated regression testing | Computer Vision | Confidence Intervals | Deep learning | Embeddings | Experimental Design | Generative AI | Human-in-the-loop | Hypothesis Testing | Image Processing | LLM | LLM VLM as a Judge | Language Models | Machine Learning | Multi-Agent | Multi-Agent Systems | Planning | PyTorch | Python | Regression testing | Reinforcement Learning | Spatial-temporal modeling | Statistics | Temporal modeling | The Loop | Tool use | VLM | Video Processing | Vision Language Models | Vision Transformers | Vision-language
Education
Roles
AI | AI Engineer | Engineer | Learning Engineer | Machine Learning Engineer
Regions
Countries
States
Cities
Related jobs
-
Software Engineer, MTIA SW Tooling Engineer USD 141K-208KC++ | CUDA | CUDA GDB | Cuda Memcheck | DWARFSenior-level Full TimeBellevue, WA | Menlo Park, CA …1h ago
-
Agent systems | Artificial Intelligence | Data Curation | Data Generation | Data PipelinesSenior-level Full TimeMenlo Park, CA1h ago
-
Production Engineer (University Grad) USD 177K-200KAI tools | API | C++ | CDN | Ethical AIMentorship | Work authorization supportSenior-level Full TimeBellevue, WA | Menlo Park, CA1h ago
-
Artificial Intelligence | Data Processing | Data debugging | Embedding Models | Fine TuningSenior-level Full TimeMountain View, CA, USA1h ago
-
Developer Relations Engineer, AI USD 138K-197KAPI Design | Debugging | Development workflows | Firebase | IDEsMid-level Full TimeNew York, NY, USA1h ago
-
Computer Vision | Data Analysis | Data Preparation | Debugging | Information RetrievalSenior-level Full TimeMountain View, CA, USA1h ago
-
Software Engineer, Model Inference, DeepMind USD 207K-300KCUDA | Caching | Distributed Systems | GPUs | Hardware accelerationSenior-level Full TimeLondon, UK; Mountain View, CA, USA1h ago
-
Software Engineer, AI/ML, Workspace USD 147K-210KData Processing | Debugging | Distributed Computing | Experimentation | Information RetrievalMid-level Full TimeSunnyvale, CA, USA; New York, NY, …1h ago
-
Senior Software Engineer, AI/ML GenAI, Core USD 174K-252KC++ | Code review | Computer Vision | Data Processing | DebuggingSenior-level Full TimeNew York, NY, USA1h ago
-
Board testing | Design for test | Machine Learning | Project Management | Risk MitigationSenior-level Full TimeSunnyvale, CA, USA1h ago
-
Acoustics | Algorithms | Android | Audio Software | C++Senior-level Full TimeMountain View, CA, USA1h ago
-
Automation | Bash | Copper cabling | Dashboard monitoring | Fiber opticsMid-level Full TimeSunnyvale, CA, USA1h ago
-
Algorithms | C++ | Data Processing | Data Structures | DebuggingSenior-level Full TimeMountain View, CA, USA1h ago
-
Lead Applied AI & Machine Learning Engineer USD 177K-215KAPI | Agentic AI | Bias Testing | Continuous integration | Data ContractsSenior-level Full TimePlano, TX, United States7h ago
-
(US) Principal ML System Engineer USD 139K-304KAWS | Access Control | Alerting | Audit Logging | AuthenticationSenior-level Full TimeRemote, USA R8h ago
-
Machine Learning Engineer, API Multicloud USD 295K-445KAPIs | AWS | Data Pipelines | Data Structures | Data structures algorithmsSenior-level Full TimeSan Francisco9h ago
-
Senior Machine Learning Engineer, Ads ML Platform USD 216K-303KAnomaly Detection | Data Governance | Data Infrastructure | Data Lineage | Data Storage401k employer match | Caregiving support | Comprehensive healthcare benefits | Family planning support | Flexible remote workSenior-level Full TimeRemote - United States R12h ago
-
Applied AI Engineer - iCloud Data USD 150K-220KAI Foundry | Apache Flink | Apache Kafka | Azure AI | Azure AI FoundrySenior-level Full TimeCupertino; Seattle12h ago
-
Applied AI Engineer - iCloud Data USD 150K-220KAWS | Agent Frameworks | Agent coordination | Apache Flink | AzureSenior-level Full TimeCupertino; Seattle12h ago
-
Applied Probability | Artificial Intelligence | C++ | Data Engineering | Data VisualizationExposure to senior leaders | Mentorship | Rotational projects | Technical trainingEntry-level Full Time141753-NC-Three Wells Fargo Center, Charlotte, United …13h ago
-
Global AI Solutions Director - Financial Services USD 200K-225KArtificial Intelligence | Cloud infrastructure | Data analytics | Data orchestration | Enterprise ArchitectureCareer development | Employee resource groups | Flexible WFH | Generous PTO | Paid volunteer timeExecutive-level Full TimeUS-New York-New York, United States R13h ago
-
Senior Software Engineer - Storage Control Plane USD 266K-395KBlock Storage | C# | C++ | CI/CD | CephSenior-level Full TimeSan Francisco Office (Fremont St) R13h ago
-
Analytics Engineer - X USD 180K-440KA/B | A/B Testing | Apache Flink | Apache Hadoop | Apache Kafka401k retirement plan | Dental insurance | Disability insurance | Employee discounts | Health insuranceMid-level Full TimePalo Alto, California13h ago
-
Human Data Manager USD 100K-186KAI Agents | Alignment Evaluation | Data Annotation | Data Generation | Data Integrity401k | Dental insurance | Disability insurance | Life insurance | Medical insuranceMid-level Full TimePalo Alto, California13h ago
-
Software Engineer, AML, Ai & Data Platforms (AiDP) USD 143K-225KCI/CD | Cassandra | Concurrency | Distributed Systems | ElasticsearchMid-level Full TimeSunnyvale13h ago