Principal AI Architect - M365 IC3 Team (Intelligent Conversation and Communications Cloud)
United States, Washington, Redmond
USD 142K-274K Senior-level Full Time
Tasks
- Build understanding of nondeterministic agent behavior
- Connect telemetry offline evaluation human judgment and experimentation signals
- Create dashboards scorecards and quality reporting
- Define AI LLM RAG agent evaluation architecture
- Design evaluation frameworks for product lifecycle
- Determine system components to address behavioral gaps
- Drive architecture decisions across services data pipelines model interfaces and search systems
- Evaluate enterprise search RAG grounding ranking permissions freshness relevance
- Identify model retrieval orchestration and quality failure modes
- Integrate ML DS LLM RAG evaluation methods into product codebases
- Integrate evals into CI CD build pipelines release gates and workflows
- Mentor engineers and applied scientists on evaluation infrastructure
- Prioritize evaluation investments with product milestones
- Stay current with LLM evaluation agent systems RAG evaluation and responsible AI
- Translate product goals into measurable evaluation strategies
Perks/Benefits
- N/A
Skills/Tech-stack
Agentic Systems | Artificial Intelligence | CI/CD | Dashboards | Data Pipelines | Data Visualization | Deep learning | Evaluation methodology | Experimentation | Grounding | Human evaluation | Indexing | Information Retrieval | LLM Observability | Language Models | Language Processing | Large Language Models | Machine Learning | Model Monitoring | Natural Language | Natural Language Processing | Offline evaluation | Online Experimentation | Permissions | Privacy | Ranking | Red Teaming | Relevance | Reliability | Retrieval-Augmented Generation | Safety | Search | Security | Synthetic data | Telemetry
Education
Roles
AI | AI Architect | Architect | Principal | Principal AI Architect
Regions
Countries
States
Related jobs
-
Research Engineer / Scientist -AI for Databases USD 162K-387KAWS | Architecture Search | Artificial Intelligence | Automated Database Tuning | Azure401k match | Conference attendance | Dental insurance | Life insurance | Long-term disabilityMid-level Full TimeSan Jose, California, United States R3h ago
-
Agent Evaluation & Evolution Machine Learning Engineer Graduate (AML-Ark-US) - 2027 Start USD 128K-256KAgent systems | Benchmarking | Data Pipelines | Deep learning | Evaluation methodology401k match | Dental insurance | Life insurance | Long-term disability | Medical insuranceEntry-level Full TimeSan Jose, California, United States4h ago
-
C++ | CUDA | Deep learning | Distributed Training | GPU AccelerationEntry-level Full TimeSan Jose, California, United States4h ago
-
Automation | Benchmarking | C++ | Compiler optimization | Data parallelismHealth insurance | Housing allowance | Life insurance | Paid Holidays | Paid sick timeEntry-level Full TimeSan Jose, California, United States4h ago
-
Abstract syntax tree | Anomaly Detection | Automated testing | C++ | Call GraphSenior-level Full TimeSan Jose, California, United States4h ago
-
Anomaly Detection | BGP | CI/CD | Cause analysis | ClickHouse401k match | Life insurance | Life insurance coverage | Long-term disability | Medical/Dental/Vision insuranceEntry-level Full TimeSan Jose, California, United States4h ago
-
Solutions Architect, Data Cloud USD 233K-324KCI/CD | Cloud Architecture | Cloud platform | Data Governance | Data ModelingSenior-level Full TimeChicago, IL, USA; Boulder, CO, USA5h ago
-
Head of Data Platform USD 176K-271KAI Governance | API Design | AWS | Access Control | Batch Processing401k employer match | Hybrid work | Life insurance | Medical/Dental/Vision insurance | Paid time offExecutive-level Full TimeHOUSTON, US, 77056 R7h ago
-
AI Software Engineering Intern USD 67K-132KAlgorithms | Data Structures | Machine Learning | Object-Oriented | Object-oriented programmingEntry-level InternshipUnited States, Washington, Redmond; United States, …9h ago
-
C# | C++ | Computer Graphics | Computer Vision | Data PipelinesEquity compensation | Health benefits | Onsite work schedule Tuesday Wednesday ThursdaySenior-level Full TimeSan Mateo, CA, United States R9h ago
-
Senior Director of Engineering, Generative AI USD 525K-573K3D Scene | 3D Scene Understanding | Agentic Systems | Artificial Intelligence | BenchmarkingEquity compensation | Health benefits | Onsite days Tuesday Wednesday ThursdaySenior-level Full TimeSan Mateo, CA, United States R9h ago
-
AI Research Resident USD 150K-150KAgent systems | Benchmarks | Constrained decoding | Dataset curation | Decision MakingCohort program | Cross-functional collaboration | Publication opportunity | Research mentorship | Startup environmentNone Full TimeNew York City10h ago
-
Analytics | Artificial Intelligence | Azure Architecture | Azure Databricks | Cloud SecuritySenior-level Full TimeUnited States11h ago
-
Staff/Principal AI Software Engineer - Snowflake CoWork USD 264K-379KAgent systems | Artificial Intelligence | Function Calling | Guardrails | LLM orchestrationSenior-level Full TimeUS-CA-Menlo Park15h ago
-
Pricipal Data Scientist USD 99K-156KData Engineering | Deep learning | Machine Learning | NumPy | PandasHealth and wellbeing resources | Paid volunteer days | Remote work flexibilityEntry-level Full TimeSan Francisco - United States - … R16h ago
-
Principal AI Software Engineer, GTM USD 194K-263KAWS | AWS CDK | AWS Lambda | Airflow | Amazon BedrockSenior-level Full TimeUnited States R16h ago
-
Sr Staff Analytics Architect USD 148K-243KAI Governance | API Development | Agent-based | Agent-based systems | Analytics architectureHybrid workSenior-level Full TimeNew York, NY, United States R17h ago
-
Applied AI Lead USD 140K-160KAI Agents | AWS | AWS Bedrock | Anthropic Claude | Automation401k match | Cell phone stipend | College savings | EAP | FSA optionsSenior-level Full TimePhiladelphia, PA, United States17h ago
-
AWS | Cloud infrastructure | Compute | Data Curation | Distributed Systems401k match | Daily lunches and snacks | Health insurance covered | Hybrid work schedule | Learning and development stipendSenior-level Full TimeRedwood City R17h ago
-
AI Research Engineer – Agentic AI USD 165K-180KAgent systems | Evaluation | Experiment design | Language Models | Large Language Models401k matching | Disability insurance | Health insurance | Life insurance | Paid time offMid-level Full TimeSunnyvale, CA, United States18h ago
-
AI/Machine Learning Engineer - Python - Loops USD 130K-150KAPI Development | CI/CD | Data Pipelines | Debugging | Embeddings401k | Community volunteering events | Dental insurance | Disability benefits | Flexible paid time offEntry-level Full TimePalo Alto, California, United States18h ago
-
AI Researcher — Intern USD 54K-70KAgentic Systems | Benchmarking | Calibration | Data Curation | Data evaluationHybrid work environment | Opportunity for publication | Research mentorshipEntry-level InternshipNew York City, NY (Hybrid) R18h ago
-
AI and Computational Geometry Engineer USD 200K-250KABAQUS | ANSYS | BREP | Boolean Operations | CAEL toolsEquity | Health benefits | Paid time offMid-level Full TimeEmeryville, California18h ago
-
Senior-level Full TimeUnited States, California, Mountain View; United …19h ago
-
Business Systems & Analytics Lead USD 163K-182KAPIs | Chart of Accounts | Cloud Data | Cloud Data Warehouse | DBTCoworking memberships | Daily meals | Flexible PTO | Health, dental, vision benefits | Learning and development stipendSenior-level Full TimeSan Francisco19h ago