Principal Data Scientist

At Microsoft, our mission is to empower every individual and organization on the planet to achieve more, and we recognize the transformative power of artificial intelligence in realizing this vision. The Bing Metrics and Evaluation team is at the forefront of this journey, meticulously measuring and enhancing the user experience across Bing and Copilot. Our goal is to execute this with precision and efficiency.

This role is available in Mountain View, CA or Redmond, WA.

In this role as a Principal Data Scientist, you will work to improve online metrics that evaluate Product quality in A/B testing and non-A/B testing situations. You will apply sophisticated statistical methods and large language models (LLMs) combined with human labels to conduct thorough evaluations of Bing’s Search Product through the lens of metrics. Your expertise will guide Bing’s product quality assessments, and user satisfaction measurement.

As part of the rapidly advancing Search and AI domain, our team is a leader in applied machine learning at Microsoft. We are committed to providing an unparalleled search and AI assistant experience to over 500 million monthly active users globally through Bing. Additionally, our work impacts other major search engines like Yahoo, DuckDuckGo, and emerging platforms such as Neeva and You.com.

Join us in shaping the future of search and AI, where your contributions will influence the experiences of users worldwide.

Responsibilities

General Product quality measurement: Define, invent, and deliver metrics that accurately measure the quality of online information and the satisfaction/success of our customers.

A/B Metrics: Design and implement A/B metrics to evaluate the impact of changes to our products and services. Analyze results to provide actionable insights and drive continuous improvement.

RAG based search experience measurement: Develop and refine metrics to assess the performance and effectiveness of our Generative SERP experience. Ensure these metrics capture user satisfaction, accuracy, and overall interaction quality.

Models: Develop ML/Statistical models to measure/predict the quality of online content and/or user interactions with large-scale AI systems.
Experimental Design: Think critically about experimental design & develop innovative strategies and products in these areas to improve the quality of A/B experimentation

Strategy: Translate strategy into plans that are clear and measurable, with progress shared out to stakeholders.

Cooperation: Partner effectively with program management, engineers, and other areas of the business.

Qualifications

Required Qualifications:

• Doctorate in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 8+ year(s) data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results)
• OR Master's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 10+ years data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results)
• OR Bachelor's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 12+ years data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results)
• OR equivalent experience.
• 5+ years experience programming experience utilizing Python, R or Big Data environments

Preferred Qualifications:

• Doctorate in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 10+ years data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results)
• OR Master's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 12+ years data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results)
• OR Bachelor's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 15+ years data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results))
• OR equivalent experience.
• Experience with Statistical modeling, A/B experimentation and building metrics for large scale user facing Artificial Intelligence products.
• Experience with human-in-the-loop machine learning.
• Experience with real world system building and data collection, including design, coding and evaluation
• Experience using LLMs (Large Language Models) to build reliable metrics & gather insights
• Customer focused, strategic, drives for results, is self-motivated, and has a propensity for action
• Fantastic problem solver: ability to solve problems that the world has not solved before

Data Science IC6 - The typical base pay range for this role across the U.S. is USD $161,600 - $286,200 per year. There is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $209,600 - $314,400 per year.

Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here: https://careers.microsoft.com/us/en/us-corporate-pay

Microsoft will accept applications for the role until September 30, 2024.

Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, ancestry, color, family or medical care leave, gender identity or expression, genetic information, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran status, race, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable laws, regulations and ordinances.  We also consider qualified applicants regardless of criminal histories, consistent with legal requirements. If you need assistance and/or a reasonable accommodation due to a disability during the application or the recruiting process, please send a request via the Accommodation request form.

Benefits/perks listed below may vary depending on the nature of your employment with Microsoft and the country where you work.