Technical AI Evaluator - Python, Automation or Security

United States - Remote R

USD 400K-400K (estimate) Mid-level Full Time

@ M...

Apply Save

Found 2d ago

Tasks

Analyze agent failures
Build simulated developer environments
Define evaluation criteria and solved conditions
Design evaluation tasks from intermediate states
Iterate tasks and tests based on QA feedback
Refine evaluation for fairness and robustness
Verify agent solutions
Write functional tests
Write integration tests

Perks/Benefits

Skills/Tech-stack

Education

N/A

Roles

Regions

North America

Countries

United States

Apply Save

Language: en Views: 2

Clicks: 0

Saves: 0

Related jobs

Featured Feat. Principal Knowledge & Data Architect USD 174K-284K

AWS Neptune | Canonicalization | Chunking | Cypher | DBT

Benefits including health and wellness programs | Health, wellness, and retirement plans

Senior-level Full Time

Headquarters - Chevy Chase, MD R

28d ago
Data Engineer USD 125K-160K

Cloud Architecture | Cloud infrastructure | Data Modeling | Data Monitoring | Data Quality

Senior-level Full Time

Tel Aviv-Yafo, Tel Aviv District, IL R

10h ago
AI Engineer for Marketing USD 96K-125K

API Security | Chroma | CrewAI | Data Governance | Data Modeling

Mid-level Full Time

Tel Aviv-Yafo, Tel Aviv District, IL R

13h ago
Developer, Applied AI, Penguin Random House (Open to Remote) USD 130K-160K

AI Search | Application Insights | Azure | Azure AI | Azure AI Search

Mid-level Full Time

New York, United States R

16h ago
Data Analytics Engineer (Puerto Rico) USD 86K-165K

APIs | Azure | CI/CD | Data Governance | Data Modeling

Achievement Awards | Career development | Child/adult backup care | Educational assistance | Flexible work schedules

Senior-level Full Time

US-PR-SANTA ISABEL-B1 ~ Felicia Industrial Park … R

23h ago
Senior Software Engineer - Machine Learning (Remote US) USD 230K-270K

AI workflows | APIs | Backend Development | Cloud Native | Context engineering

Remote work

Senior-level Full Time

USA R

23h ago
Embedded Controls Software Engineer USD 85K-192K

ASPICE | AUTOSAR | C# | C++ | CAN

Flexible family care days | Medical, dental & vision coverage | Paid Holidays | Paid parental leave | Paid time off

Senior-level Full Time

Dearborn, MI, United States R

1d ago
Lead Machine Learning Engineer USD 172K-276K

AWS | Azure | CI/CD | Databricks | Distributed Systems

Flexible autonomy | Learning and development programs | Mentorship

Senior-level Full Time

Chicago, Illinois, USA R

1d ago
Developer, Applied AI, Penguin Random House (Open to Remote) USD 130K-160K

Application Insights | Azure | Azure Monitor | BeautifulSoup | CI/CD

401k | Dental insurance | Educational assistance | Health insurance | Open to remote

Mid-level Full Time

New York, United States R

1d ago
Senior Full Stack AI Engineer USD 150K-176K

AWS | Agents | Azure | CI/CD | Embeddings

401k matching | Career growth opportunities | Commuter benefits | Dental coverage | HSA & FSA options

Senior-level Full Time

United States R

1d ago
Sr Machine Learning Engineer I USD 150K-241K

Automated testing | CI/CD | Data Versioning | Experimental Design | MLOps

Discretionary paid time off | Emotional & mental wellness support | Employee resource groups | Fitness programs | Learning and development programs

Senior-level Full Time

Seattle, Washington, United States R

1d ago
Principal Engineer, Data & ML Infrastructure USD 200K-275K

AWS | Cloud Computing | Data Mining | Data Processing | Data Warehousing

401k match | Dental insurance | Health savings account | Life insurance | Medical insurance

Senior-level Full Time

Boston, Massachusetts, United States; Pittsburgh, Pennsylvania, … R

1d ago
Data Analytics Engineer USD 139K-230K

AWS | Azure | Azure Data | Azure Data Lake | Cloud platform

DEI initiatives | Dental insurance | Medical insurance | Mental health support | Remote work

Senior-level Full Time

Remote, United States R

1d ago
Data Engineer USD 126K-208K

API Integration | APIs | AWS | Amazon Redshift | Apache Airflow

DEI initiatives | Dental insurance | Employee recognition program | Medical insurance | Mental health support

Mid-level Full Time

Remote, United States R

1d ago
Data Engineer (Healthcare) USD 105K-130K

AWS | AWS Redshift | Apache Airflow | Azure | Azure Data

Knowledge sharing | Mentoring | Quarterly travel | Remote work

Mid-level Full Time

USA, Remote; USA, Remote; USA, Remote R

1d ago
Staff Software Engineer, AI Foundations (AI Agent Optimization) CAD 224K-302K

API Design | Atomics | Concurrency | Distributed Systems | Generative AI

Home office setup | Internet stipend | Learning and development budget | Paid Holidays | Professional memberships

Senior-level Full Time

United States or Canada, Remote Opportunity R

1d ago
Applied AI Solutions Architect USD 156K-209K

AWS Bedrock | AWS SageMaker | Agentic Architectures | Anthropic | Azure AI

401k company match | Dental insurance | Home office & equipment stipend | Learning and development stipend | Paid Holidays

Senior-level Full Time

US - Remote R

1d ago
Senior AI Product Builder, Inventory Optimization USD 151K-251K

API Design | Artificial Intelligence | Data Contracts | Data Modeling | Demand forecasting

401k match | Basic life insurance | Flexible time off | Medical/Dental/Vision insurance | Paid Holidays

Senior-level Full Time

Remote - United States R

1d ago
Senior Machine Learning (ML) Engineer (AI Insurtech) USD 190K-225K

BigQuery | Dagster | Data Analysis | Data Modeling | Exploratory Data Analysis

401k matching | ADandD insurance | Catered lunches | Dental insurance | Family benefits

Senior-level Full Time

New York, NY or Remote R

1d ago
Data Engineer (GCP, BigQuery) CAD 70K-120K

ANSI SQL | Agile | BigQuery | Cloud platform | Clustering

Birthday bonus | Dental insurance | Health & wellness days | Holiday time | Medical insurance

Mid-level Full Time

United States R

1d ago
Associate Data and Analytics Engineer USD 62K-76K

AWS | Access Control | Alerting | Data Governance | Data Modeling

Mid-level Full Time

Blacksburg, Virginia, Fully Remote, Hybrid R

1d ago
Senior HPC and Quantum Systems Engineer USD 224K-431K

API Integration | C++ | CUDA | CUDA-Q | CuQuantum

Senior-level Full Time

US, MA, Westford R

1d ago
Data Engineer (2258) USD 119K-150K

AWS | Azure | Data Visualization | Docker | ETL

401k | Dependent care | Discounted tickets and entertainment perks | Flexible work schedule | Medical insurance

Mid-level Full Time

New York, New York, United States R

1d ago
Senior AI Engineer USD 122K-160K

APIs | AWS | Asynchronous programming | Autogen | Caching

Senior-level Full Time

USA R

1d ago
Head of AI/ML (Director/VP) - VC Backed Startups USD 250K-300K

AWS | Airflow | Artificial Intelligence | Azure | Data Engineering

Executive-level Full Time

NYC - Hybrid R

1d ago