Senior Data Scientist, AI Product Performance

Are you passionate about building intelligent systems that respond to users in real time? Do you thrive in environments where data-driven experimentation and product analytics are core to the product development process?

At Xbox Gaming AI, we are tapping into the power of generative AI to power the next generation of Xbox experiences. We are venturing beyond the horizon and charting a course forward with players and creators at the center. Our goal is to define the future of Xbox by advancing our mission of bringing joy and community to every player on the planet.

We’re looking for a Senior Data Scientist, AI Product Performance with a deep focus on end-to-end AI product analytics who can help us rapidly and iteratively unlock the power of real-time user signals to drive retention, product quality & success. This role will drive instrumentation strategy through deep, causal insights – so our team can ship smarter, faster and build higher value experiences.

This role is for someone who is comfortable working across the full data lifecycle—from designing metrics and pipelines to performing detailed analysis and delivering insights that directly shape product decisions.
You’ll work closely with product, engineering, and design teams to define how people interact with and play games. We’re looking for someone with analytical rigor, high quality technical skills, and a deep understanding of user behavior—paired with a creative mindset and high ownership.

Your charter: turn event streams, A/B tests, logs and user research into the signals that shape roadmap, model iterations and growth loops.

Responsibilities

• Establish standardized scorecard evaluation metrics for AI products, covering user engagement, model performance, and product quality, to enable consistent tracking across teams and disciplines
• Implement robust data pipelines and evaluation calculations by leveraging tools such as Python, C#, SQL/KQL, Power BI, and LLM prompting techniques; partner with engineering teams to ensure solutions are scalable, reliable, and seamlessly integrated into product workflows for both real-time and offline testing scenarios
• Deliver actionable and compelling insights by deep diving into analytics to measure user acquisition, activation, engagement, retention, churn, and overall ecosystem health; identify key patterns and feedback loops to inform product and model improvements, and drive prioritization.
• Analyze conversational product interactions—including language content, semantics, and dialogue structure—to uncover user intent and behavior; apply topic clustering, semantic embedding techniques, and build lightweight classifiers to model usage patterns
• Effectively present and communicate product data insights through dashboards, written reports, and stakeholder discussions, enabling teams to understand user behavior and product performance through both qualitative and quantitative lenses
• Transform ad hoc analyses into scalable, maintainable pipelines and reusable analytics functions; automate data preparation workflows and support ML experimentation by generating high-quality training and evaluation datasets
• Develop and apply offline and online evaluation methods—including model regression tests, A/B testing, and real-world usage tracking; train and deploy classifiers or scoring models to evaluate AI output quality (e.g., relevance, helpfulness, coherence) at scale

Qualifications

Required Qualifications (RQs) 

• Doctorate in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 1+ year(s) data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results)
  • OR Master's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 3+ years data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results)
  • OR Bachelor's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 5+ years data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results)
  • OR equivalent experience.

• 3+ year(s) building or supporting a consumer product and Experience guiding product roadmaps with data.
• 3+ Proficiency in SQL and Python (or R), with experience using common data science libraries and tools
• 3+ Experience with real-time data systems and large-scale product data (e.g., Kafka, Azure Data Explorer), ML infrastructure for evaluation and deployment, and building lightweight classifiers or recommendation components (e.g., retrieval, ranking, embeddings, personalization) to support AI product performance

Preferred Qualifications (PQs)

• Ability to work in ambiguous problem spaces and design analytical frameworks from the ground up
• Experience supporting or building evaluation systems for language models, recommendation engines, or personalization systems
• Experience with analytics in digital consumer products, particularly AI/LLM-driven features, including experiment design and casual inference analysis

Data Science IC4 - The typical base pay range for this role across Canada is CAD $114,400 - CAD $203,900 per year.

Find additional pay information here:

https://careers.microsoft.com/v2/global/en/canada-pay-information.html

Microsoft will accept applications for the role until August 11, 2025

Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, ancestry, color, family or medical care leave, gender identity or expression, genetic information, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran status, race, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable laws, regulations and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or the recruiting process, please send a request via the Accommodation request form.

Benefits/perks listed below may vary depending on the nature of your employment with Microsoft and the country where you work.