Software Engineer in Test, LLM AI & Automation Focus

POSITION LOCATION: San Carlos, CA, Austin, TX, or Remote USA   POSITION SUMMARY:
We are seeking a Software Engineer in Test ( AI & Automation Focus) to join a forward-thinking engineering team responsible for building and evaluating AI-powered systems, including a sophisticated voice assistant platform to help patients get the help they need fast. This is not a traditional SQE role - this is a unique opportunity to blend test automation, machine learning evaluation, and data analysis into one impactful position.
You'll design test automation frameworks, evaluate large language model (LLM) performance, and develop custom tooling to analyze, monitor, and continuously improve product quality in both traditional software and AI-driven components. KEY RESPONSIBILITIES:

• Design, build, and maintain robust, data-driven test automation frameworks JavaScript/TypeScript and Python.
• Build automated evaluation pipelines and prompt-based tests for LLM and conversational AI systems.
• Analyze large-scale, unstructured logs and telemetry data to identify issues and user experience gaps.
• Create custom scripts to support test automation, log parsing, and performance evaluation of machine learning models.
• Develop visual dashboards and reports to track quality, system health, and model behavior over time.
• Support test environment setup and integrate with CI/CD pipelines for continuous testing.
• Perform root cause analysis, debugging, and reporting of product issues, collaborating closely with Developers and SQEs across the SDLC.
• Drive innovation by introducing new testing methodologies, tools, and best practices for both deterministic and non-deterministic systems.
• Partner with global, cross-functional teams including ML engineers, product managers, and infrastructure leads.

QUALIFICATIONS:

• BS degree in Computer Science or related technical field.
• 3+ years of experience in test automation, ML evaluation, data analysis, or related roles.

KNOWLEDGE, SKILLS, AND ABILITIES:

• Proficient in JavaScript/TypeScript and Python; strong scripting experience, especially for automation and log parsing.
• Experience with tools such as Jest, Playwright, Cypress, or Selenium for UI and API automation.
• Strong knowledge of SQE principles, risk-based testing, test planning, and system integration.
• Knowledge of testing approaches for AI based products, such as AI evaluations metrics and their usage.
• Familiarity with LLMs, GenAI, and prompt engineering .
• Comfortable working with API testing, log data analysis, and performance metrics.
• Experience in evaluating and testing non-deterministic systems AI-powered projects such as conversational models or recommendation engines.
• Solid understanding of CI/CD pipelines, cloud-based environments, and microservices/serverless architectures.
• Strong communication and analytical skills; ability to clearly convey technical findings to both technical and non-technical audiences.
• Self-starter with a growth mindset and a passion for building scalable, intelligent testing systems.

  The pay range is listed and actual compensation packages are based on a wide array of factors unique to each candidate, including but not limited to skill set, years & depth of experience, certifications and specific office location. This may differ in other locations due to cost of labor considerations.Remote USA$94,300—$117,900 USD

OUR OPPORTUNITY

Natera™ is a global leader in cell-free DNA (cfDNA) testing, dedicated to oncology, women’s health, and organ health. Our aim is to make personalized genetic testing and diagnostics part of the standard of care to protect health and enable earlier and more targeted interventions that lead to longer, healthier lives.

The Natera team consists of highly dedicated statisticians, geneticists, doctors, laboratory scientists, business professionals, software engineers and many other professionals from world-class institutions, who care deeply for our work and each other. When you join Natera, you’ll work hard and grow quickly. Working alongside the elite of the industry, you’ll be stretched and challenged, and take pride in being part of a company that is changing the landscape of genetic disease management.

WHAT WE OFFER

Competitive Benefits - Employee benefits include comprehensive medical, dental, vision, life and disability plans for eligible employees and their dependents. Additionally, Natera employees and their immediate families receive free testing in addition to fertility care benefits. Other benefits include pregnancy and baby bonding leave, 401k benefits, commuter benefits and much more. We also offer a generous employee referral program!

For more information, visit www.natera.com.

Natera is proud to be an Equal Opportunity Employer. We are committed to ensuring a diverse and inclusive workplace environment, and welcome people of different backgrounds, experiences, abilities and perspectives. Inclusive collaboration benefits our employees, our community and our patients, and is critical to our mission of changing the management of disease worldwide.

All qualified applicants are encouraged to apply, and will be considered without regard to race, color, religion, gender, gender identity or expression, sexual orientation, national origin, genetics, age, veteran status, disability or any other legally protected status. We also consider qualified applicants regardless of criminal histories, consistent with applicable laws.

If you are based in California, we encourage you to read this important information for California residents. 

Link: https://www.natera.com/notice-of-data-collection-california-residents/

Please be advised that Natera will reach out to candidates with a @natera.com email domain ONLY. Email communications from all other domain names are not from Natera or its employees and are fraudulent. Natera does not request interviews via text messages and does not ask for personal information until a candidate has engaged with the company and has spoken to a recruiter and the hiring team. Natera takes cyber crimes seriously, and will collaborate with law enforcement authorities to prosecute any related cyber crimes.

For more information:
- BBB announcement on job scams 
- FBI Cyber Crime resource page