Director, Model Behavior & Evaluation Systems

Tasks

• Align prompts tools APIs and workflows with intended behavior
• Build high performing evaluation governance and SME review team
• Build model behavior evaluation systems
• Communicate behavior strategy and quality tradeoffs
• Convert traces and failures into behavior specs
• Create scenario coverage and eval datasets
• Define behavior launch criteria
• Establish evaluation release gates
• Lead red teaming and behavioral risk programs
• Own behavioral quality bar for LLM agents
• Partner on reinforcement learning and post training data
• Review model traces and evaluation outputs for launch readiness
• Set operating cadence for behavior reviews and monitoring
• Set technical direction for behavior and evaluation infrastructure

Perks/Benefits

• N/A

Skills/Tech-stack

API Integration | Agents SDK | Fine Tuning | Hugging Face | Hugging Face Datasets | Hugging Face Transformers | Inference Optimization | LLM Evaluation | Langfuse | Model checkpoints | NVIDIA NeMo RL | NVIDIA Nemo | NeMo-RL | Observability | OpenAI Agents | OpenAI Agents SDK | Preference data | PyTorch | Python | RLAIF | RLHF | Red Teaming | Regression testing | Reinforcement Learning | Reward Modeling | Supervised Fine Tuning | Synthetic data | Tokenization | Tool-Calling | Trace analysis