Senior AI Engineer - Observability
Tasks
- Analyze production traces to identify regressions and cost spikes
- Build and improve LLM powered product systems
- Build dashboards alerts and runbooks for failures
- Create and maintain golden datasets for LLM features
- Define quality gates and ship readiness thresholds
- Design evaluation pipelines and scoring methodologies
- Develop reusable libraries SDKs and templates for evaluation and observability
- Document standards for tracing metadata and evaluation reporting
- Ensure security privacy SOC 2 ISO 27001 and data residency compliance
- Establish prompt and retrieval regression testing
- Implement LLM judge and quality scoring
- Instrument LLM interactions and RAG pipelines
- Monitor guardrails and policy enforcement signals
- Monitor latency token usage cost retrieval quality and safety signals
Perks/Benefits
- N/A
Skills/Tech-stack
.NET | A/B | A/B Testing | AI Search | Agentic Workflows | Alerting | Application Insights | Arize | Azure AI | Azure AI Search | Azure DevOps | Azure Monitor | B testing | CI/CD | Context Assembly | Csharp | Embeddings | Evaluation | Git | Groundedness | Hallucination detection | Human Feedback | Humanloop | ISO 27001 | LLM | LLMOps | Langchain | Langfuse | Langsmith | Llamaindex | Logging | Metrics | Observability | Online Experimentation | OpenTelemetry | PGVector | PII handling | Pinecone | Prompt engineering | Python | Qdrant | RAG | Ranking | Retrieval Quality | SOC 2 | Semantic Search | Tracing | Vector Search | WandB | Weaviate
Education
N/A
Related jobs
-
Senior Machine Learning Engineer USD 160K-200KAWS | CI/CD | Computer Vision | Data Processing | DockerCommuter benefits | Dental insurance | Employee assistance program | Equity | Health insuranceSenior-level Full TimeArlington, VA R22h ago
-
Sr Staff AI Engineer - Veza USD 190K-334KAgent Orchestration | Algorithms | C++ | Data Structures | Distributed Systems401k plan with match | ESPP | Family leave programs | Flexible time away | Health plansSenior-level Full TimeMinneapolis, Minnesota, United States R22h ago
-
AI/ML Engineer USD 152K-220K3D Reconstruction | 3D Scene | 3D Scene Understanding | AI Agents | Cloud ComputingAgile team collaboration | Fully remote | Production monitoring ownershipMid-level Full TimeSan Francisco, CA (Hybrid) or United … R23h ago
-
Senior GTM AI Engineer USD 154K-213KAgent Orchestration | Agent Performance | Agent Performance Monitoring | Anthropic Claude | App Development401k match | Catered lunch | Company events | Desk setup stipend | Flexible spending accountsSenior-level Full TimeOakland, California, United States R23h ago
-
Data Science Engineer USD 121K-154KData Visualization | Documentation | Entity Resolution | Fine Tuning | Git401k | Education reimbursement program | Flexible schedules | Hybrid schedule | Relocation assistanceEntry-level Full TimeLivermore, CA, United States R23h ago
-
Senior Analytics Engineer USD 153K-206KAWS | Data Quality | ETL | Healthcare Claims | PySpark401k plan | Commuter benefits | Dental insurance | Disability insurance | Flexible spending accountSenior-level Full TimeRemote- US R1d ago
-
Senior AI Analytics Engineer USD 149K-212KAI Agents | Agent Frameworks | Automation | DBT | Python401k match | Employee assistance program | HSA contribution | Health insurance | Hybrid workSenior-level Full TimePlano,Texas,United States R1d ago
-
800-37 | 800-53 | AI Governance | AI orchestration | AI red teaming401k | Dental insurance | Disability insurance | Employee assistance program | Hybrid work scheduleSenior-level Full TimeWashington, DC R1d ago
-
IAA - Principal Data Engineer USD 171K-256KAzure | Azure Synapse | Azure Synapse Analytics | CI/CD | Data GovernanceHybrid work schedule | Onsite expectation 2 days per weekSenior-level Full TimeChicago, IL (Hybrid) R1d ago
-
Senior Analytics Engineer USD 149K-195KAWS | Alerting | Azure | Cloud Data | Cloud Data PlatformsAnnual performance bonuses | Company equity grants | Hybrid work schedule | Unlimited vacationSenior-level Full TimeNew York, New York, United States R1d ago
-
Full-Stack AI Engineer USD 150K-200KAWS | Authentication | Azure | CI/CD | CSSD E And I Initiatives | Fertility and menopause leave | Fitness allowance | Health benefits | Hybrid workingMid-level Full TimeNew York R1d ago
-
Software Engineer I USD 97K-122KBatch inference | Cloud infrastructure | Docker | GPU Optimization | GoConference reimbursement | Education reimbursement | Employee assistance program | Flexible time off | Hybrid workEntry-level Full TimeSeattle R1d ago
-
Applied AI Architect, Startups USD 240K-315KAI architecture | Context engineering | Integration Patterns | LLM Evaluation | Language ModelsFlexible working hours | Generous vacation | Optional equity donation matching | Parental leaveSenior-level Full TimeSan Francisco, CA | New York … R1d ago
-
Data Scientist USD 160K-170KAPI Integration | APIs | Agentic Workflows | Anomaly Detection | Artificial Intelligence401k retirement plan | Employee assistance program | Health Savings Account (HSA) | Life insurance | Long-term disabilityMid-level ContractPhoenix, Arizona, USA R1d ago
-
Mid-level Full TimeRemote, US R1d ago
-
Data Engineer USD 153K-207KAirflow | Automated testing | CI/CD | DBT | Data GovernanceFlexible hybrid schedule | Hybrid work arrangement | Onsite Days Per WeekMid-level Full TimeSan Francisco, CA R1d ago
-
AWS | Agent systems | Apache Spark | Azure | Cloud platformOn site customer collaboration | Remote work | Travel for customer visitsSenior-level Full TimeWashington, D.C. R1d ago
-
AWS | Agent systems | Apache Spark | Apache Spark SQL | AzureConference presentations | Periodic onsite work | Remote work possible | Travel every 4 to 8 weeksSenior-level Full TimeVirginia R1d ago
-
AI Integrations Staff Engineer USD 150K-230KAI Models | Agent SDKs | Back-end | Back-end development | Data ModelingAutomatic 401k contribution | Bi-annual company retreats | Employee referral program | Equity | Home office setupSenior-level Full TimeRemote R1d ago
-
AI Security Engineer USD 150K-160KAPI Gateway | AWS | Adversarial Testing | Azure | CI/CD401k retirement plan | Employee assistance program | Health savings account | Life insurance | Medical, dental, and visionMid-level ContractBolingbrook, Illinois, USA R1d ago
-
.NET | AI Foundry | AI Services | Action Planning | Agent FrameworkContract-to-hire | GC sponsorship available | Hybrid work | US C sponsorship availableSenior-level ContractDallas, United States R1d ago
-
Lead Software Engineer (.Net Core + LLM's) USD 150K-225K.NET | AKS | ASP.Net Core | Airflow | Application Insights401k matching | Childcare assistance | Dental insurance | Disability insurance | Education reimbursementSenior-level Full TimeDallas, Texas, United States R1d ago
-
Senior Data Platform Engineer USD 180K-220KAKS | Airflow | Apache Spark | Azure | Azure SQL401k matching | Childcare assistance | Dental insurance | Disability insurance | Education reimbursementSenior-level Full TimeDallas, Texas, United States R1d ago
-
Argo CD | ArgoCD | CSI | CSI driver | FluxExecutive-level Full TimeRemote, USA, United States R1d ago
-
AI/ML Engineer USD 100K-187KAWS | Algorithms | Computer Vision | Data Structures | Deep learning401k | Annual bonus | Apple hardware ecosystem | Dental insurance | Disability insuranceMid-level Full TimeSan Francisco R1d ago