1033- AI Engineer (Evaluation Systems)
Tasks
- Build calibration workflows
- Combine deterministic checks with LLM verdicts
- Conduct A/B testing
- Create error analysis pipelines
- Curate evaluation data
- Design evaluation engine
- Develop specialized judges
- Handle delayed outcomes and low-confidence flows
- Measure precision and recall
- Monitor drift and trigger recalibration
- Perform content-level evaluation
- Store structured verdicts
Perks/Benefits
Skills/Tech-stack
AWS | Anthropic | Asynchronous programming | Docker | Evaluation Pipelines | LLM APIs | ML Engineering | Model Performance Assessment | Model performance | OpenAI | Performance Assessment | PostgreSQL | Python
Education
N/A
Roles
AI | AI Engineer | Engineer
Related jobs
-
AI Engineer MXN 198K-233KAPI Integration | AWS | CI/CD | Containerization | Data pipelineDiscounts | English classes | Flexible schedule | Health insurance | Holidays and vacationsEntry-level Full TimeDinastía, Nuevo León, Mexico - Remote R15h ago
-
Senior Machine Learning Engineer USD 120K-195KAnomaly Detection | Apache Beam | Apache Spark | BigQuery | CI/CD401k | Compassionate leave | Dental insurance | Discount Portal | HolidaysSenior-level Full TimeRemote - US R17h ago
-
AI Engineer (Hybrid @ Bellevue, WA or Remote @ Florida) USD 140K-160KAWS | Artificial Intelligence | Asyncio | Evals | Function Calling401k match | Disability benefits | Family leave | Health insurance | Life insuranceEntry-level Full TimeHybrid @ Bellevue, WA or Remote … R18h ago
-
Databricks Data Engineer USD 166K-247KAWS EC2 | AWS Glue | AWS Lambda | AWS Step Functions | Agile401k | Dental insurance | Disability insurance | Educational assistance | Gym reimbursementMid-level Full TimeREMOTE R20h ago
-
Senior MLOps Platform Engineer {S} USD 120K-185KAWS EKS | Airflow | Amazon S3 | Argo CD | Batching401k match | Dental insurance | Employee assistance program | HSA contributions | Health insuranceSenior-level Full TimeColorado Springs, Colorado, United States R20h ago
-
Senior Machine Learning Engineer USD 108K-160KA/B | A/B Testing | AWS Lambda | Algolia | Amazon OpenSearch401k | Dental insurance | Disability insurance | Life insurance | Medical insuranceSenior-level Full TimeRemote US R20h ago
-
Data Engineer USD 124K-166KAPI Security | AWS | AWS Cloud | AWS Cloud Security | Amazon Kinesis401k retirement plan | Company paid premiums | Conference reimbursement | Dental insurance | Disability insuranceSenior-level Full TimeUnited States - Remote R21h ago
-
Senior Staff Engineer - Data Scientist USD 150K-190KAmazon Web Services | Anomaly Detection | Apache Airflow | Apache Kafka | Apache SparkContract-to-hire | Remote work | Travel opportunitiesSenior-level Full TimeRemote, REMOTE, United States R21h ago
-
Manager, Data Engineering USD 130K-166KAWS | Access Controls | Apache Airflow | Audit Logging | AzureCollaborative team culture | Remote work | Work-life balanceSenior-level Full TimeRemote, United States R22h ago
-
Senior Software Engineer - Data Platform USD 150K-195KAPIs | BigQuery | C++ | CI/CD | Cloud Storage401k company contribution | Disability insurance | Fertility and infertility benefits | Industry competitive PTO | Learning and development opportunitiesSenior-level Full TimeUnited States R22h ago
-
AI Safety | API Development | Context Management | Context window | Context window optimizationDiversity and inclusion | Fitness options | Flexible work-life balance | Health support | Professional developmentSenior-level Full TimeRomania R22h ago
-
API Integration | Agent Frameworks | Artificial Intelligence | Backend Services | Content ModerationCollaborative team environment | High impact AI applications | Modern development tools and resources | Remote work flexibilitySenior-level Full TimeCanada R22h ago
-
AWS | Agile | Azure | Cloud Computing | Data Governance100 percent remote | Career growth | Continuous learning | Innovation culture | International projectsSenior-level Full TimeCanada R22h ago
-
Mid-level Full TimeGuadalajara Mexico R23h ago
-
Software Engineer MXN 342K-420KAWS | Alerting | Cause analysis | Data Modeling | Database Performance OptimizationMid-level Full TimeGuadalajara Mexico R23h ago
-
Mid-level Full TimeGuadalajara Mexico R23h ago
-
Mid-level Full TimeGuadalajara Mexico R23h ago
-
GLiNER | Knowledge graphs | Language Models | Language Processing | Large Language ModelsEquity | Remote work | Technical ownershipSenior-level Full TimeAustria R23h ago
-
.NET | AI Foundry | AI Search | Agent Framework | Agent OrchestrationCareer development | Flexible work environment | Friendly collaborative team culture | Professional developmentSenior-level Full TimeAustria R23h ago
-
.NET | AWS | Azure | C# | Distributed Systems401k match | Generous PTO plan | Medical, dental, vision benefitsSenior-level Full TimeAustria R23h ago
-
Senior Data Platform Engineer (Python/AWS) - Finance USD 132K-174KAWS CDK | AWS CloudFormation | AWS Glue | AWS Lambda | AWS Step FunctionsPaid time off | Remote work | Work autonomySenior-level Full TimeSanto Domingo R23h ago
-
Frontend Engineer, Forward Deployed (GenAI) USD 147K-216KAg Grid | Algorithms | Azure OpenAI | Claude API | Data StructuresHybrid work environment | Onsite client collaboration 3 to 4 days per weekSenior-level Full TimeNew York - Hybrid R23h ago
-
Senior Data Platform Engineer (Python/AWS) - Finance MXN 1040K-1410KAWS CDK | AWS CloudFormation | AWS Glue | AWS Lambda | AWS Step FunctionsPaid time off | Remote work | Work autonomySenior-level Full TimeMexico City R23h ago
-
Senior Data Platform Engineer (Python/AWS) - Finance BRL 258K-360KAWS CDK | AWS Glue | AWS Lambda | AWS Step Functions | Amazon ECSPaid time off | Remote work | Work autonomy | Work with top US companiesSenior-level Full TimeSão Paulo R23h ago
-
Senior Software Engineer MXN 1040K-1300KAWS | Alerting | Cause analysis | Data Ingestion | Data ModelingSenior-level Full TimeGuadalajara, Mexico R23h ago