Senior AI Backend Engineer - Agent Evaluation & Quality
Makkah, Makkah Province, Saudi Arabia - Remote
R
USD 175K-250K (estimate) Senior-level Full Time
Tasks
- Build user simulators for test coverage
- Calibrate models with human labels
- Contribute to agent development
- Create per PR eval harnesses
- Define measurable quality criteria
- Design LLM as judge systems
- Generate adversarial test cases
- Implement regression detection in CI
- Integrate production failures into evaluation sets
- Own evaluation stack
Perks/Benefits
- N/A
Skills/Tech-stack
API Design | CI/CD | Cost Optimization | Experimentation | Function Calling | LLM Evaluation | LLM-as-judge | Langchain | Langgraph | Latency | Observability | Orchestration frameworks | Python | RAG | Regression testing | Reliability | System design | Testing | Tool use | TypeScript
Education
N/A
Roles
AI | AI Backend Engineer | Backend Engineer | Engineer | Senior AI Backend Engineer
Related jobs
-
Alerting | Amazon Redshift | CI/CD | Cause analysis | DashboardsContinuous improvement | Cross-functional collaboration | Fully remote work | Opportunity for ownershipSenior-level Full TimeSaudi Arabia R22h ago
-
AI Infrastructure Engineer III USD 152K-276KAWS | Ansible | Azure | Bash | CI/CDAutonomy | Flexible responsibility | Fun dynamic workplace | Health insuranceSenior-level Full TimeRemote-Egypt, Saudi Arabia R1d ago
-
Adjudication | Annotation Guidelines | Annotation platforms | Audio evaluation | CalibrationFully remote | International team collaboration | Professional growth opportunitiesSenior-level Full TimeSaudi Arabia R7d ago
-
Senior-level Contract Full TimeRiyadh, Riyadh Province, Saudi Arabia - … R12d ago
-
AWS | Ansible | Backpressure | Backup and Recovery | BashAutonomy and responsibility | Collaborative culture | Fun dynamic workplace | Top-tier health insuranceSenior-level Full TimeRemote-Egypt, Saudi Arabia R20d ago