Senior AI Backend Engineer - Agent Evaluation & Quality

Tasks

• Build user simulators for test coverage
• Calibrate models with human labels
• Contribute to agent development
• Create per PR eval harnesses
• Define measurable quality criteria
• Design LLM as judge systems
• Generate adversarial test cases
• Implement regression detection in CI
• Integrate production failures into evaluation sets
• Own evaluation stack

Perks/Benefits

• N/A

Skills/Tech-stack

API Design | CI/CD | Cost Optimization | Experimentation | Function Calling | LLM Evaluation | LLM-as-judge | Langchain | Langgraph | Latency | Observability | Orchestration frameworks | Python | RAG | Regression testing | Reliability | System design | Testing | Tool use | TypeScript