Senior ML Ops Engineer

About Marshmallow

We build financial products that accelerate the economic freedom for people who move across borders. We started with car insurance — insuring over a million drivers — and we’re scaling beyond. Tens of millions of people move countries each year, facing overlooked financial challenges. Our future is in building financial products around their needs to positively impact their lives.

How we work

We’re really proud of the culture we’ve created. We push for progress every day, because we know that we’ll only hit big milestones by taking lots of smaller steps. We’re always open to helping our team mates, sharing our ideas, experience and knowledge to solve problems together. We take risks, think creatively and experiment relentlessly to meet our customer’s needs, and never pass blame when things go wrong. We encourage people at all levels to take ownership of their work, and to be bold in challenging how we do things. Everyone has a voice and the opportunity to make an impact.
And autonomy and ownership are only possible with clear direction. That’s why we collaborate to do in-depth planning twice a year, and make sure we leave with clear goals and objectives that flow from top to bottom. To make sure we’re as aligned as possible across functions, most of our work rolls up into three tribes; Acquisition, Retention & Claims. Each tribe has multiple teams embedded in it, working cross-functionally to do great work.
We’re so excited for all of the challenges up ahead, and we need more people to help us tackle them! If life at Marshmallow sounds like it could be for you, explore our Culture Handbook to find out more.

The Data Science Team

Our central Data Science & AI team owns the end‑to‑end lifecycle of every model at Marshmallow, from experimentation to production. As an insurer‑fintech serving hundreds of thousands of customers, reliability and speed are critical - our models quote prices, detect fraud and power next‑gen GenAI experiences in real time.

As a Senior ML Ops Engineer, you'll play a crucial role as an individual contributor, working closely alongside data scientists and platform engineers to ensure our models are deployed quickly, safely, and at scale.

What you’ll be doing

• Designing, building, and continuously enhancing ML Ops pipelines to ensure model iteration cycles of less than 24 hours and over 99.9% production uptime.

• Managing and improving multiple deployment pipelines using AWS SageMaker (real-time and batch inference), integrated with TeamCity for seamless CI/CD.

• Collaborating closely with data scientists to containerise experiments, automate feature generation, and efficiently transition models from notebooks to production environments.

• Developing comprehensive monitoring and observability solutions to proactively detect performance drift, data-quality issues, and model performance anomalies, integrating alerts into our incident-management workflows.

• Exploring and evaluating emerging GenAI and LLM tools to accelerate the delivery of new use cases.

• Partnering with platform engineers to embed infrastructure-as-code (Terraform) and security best practices into every release.

• Enhancing the current MLOps tool stack by introducing new tools to the organisation to advance our overall MLOps maturity.

• Participating in strategic evaluations around build-versus-buy decisions and vendor integrations, prioritising cost-effectiveness and a superior developer experience.

• Promoting a “You build it, you run it” culture by informally mentoring peers and setting high technical standards, without formal managerial responsibilities.

• Contributing to the development and shaping of our MLOps/LLMOps vision and strategic roadmap for the future

Who you are

You are a practical engineer passionate about deploying robust and reliable ML systems. You thrive on solving complex infrastructure challenges, automating manual tasks, and empowering data scientists. You lead through deep technical knowledge, flourish in dynamic environments characterised by ownership and ambiguity, and consistently stay informed about the evolving ML and GenAI landscape.

What we're looking for from you

Experiences that are essential

• At least 4 years of hands-on ML Ops or ML platform engineering experience in production environments (experience in finance, insurance, or similarly regulated industries is advantageous).

• Proven track record delivering reliable CI/CD pipelines for ML workloads (using tools like TeamCity, GitHub Actions, Jenkins, etc.).

• Extensive production experience with AWS, particularly SageMaker for model training and deployment.

• Strong Python programming skills with familiarity in integrating at least one backend language (e.g., Java).

• Expertise in infrastructure-as-code tools (e.g., Terraform) and containerisation technologies (Docker).

• Experience designing robust monitoring solutions to detect model drift, performance issues, and data-quality anomalies.

• Demonstrated ability to influence and collaborate effectively with cross-functional teams without direct managerial authority.

Experiences that will help you

• Working with feature stores such as Tecton.

• Implementing or integrating model‑monitoring platforms like Arize.

• Deploying or operating LLM‑based applications at scale.

• Negotiating with vendors and managing third‑party contracts.

• Familiarity with Kubernetes or serverless architectures on AWS.

• An understanding of FCA compliance considerations for ML services.

• Comfortable with vector DBS, embedding management and model lifecycle tooling.

• Experience with model parallelism, distributed training.

• Experience with developing CI/CD pipelines within TeamCity using Kotlin.

Perks of the job

• Flexi-office working – Spend 2-3 days a week with your team in our new collaborative London office. The rest is up to you! 🏠

• Competitive bonus scheme - designed to reward and recognise high performance 🌟

• Flexible benefits budget - £50 per month to spend on a Ben Mastercard, meaning you get your own benefits budget to spend on things you want. Whether that’s subscriptions, night classes (puppy yoga, anyone?), the big shop or a forest of houseplants. Pretty much anything goes 💰

• Sabbatical Leave - Get a 4-week fully paid sabbatical after being with us for 4 years 🏝️

• Work From Anywhere - 4 weeks work from anywhere to use, with no need to come to the office 🛫

• Mental wellbeing support – Access therapy and mental health sessions through Oliva 💚

• Learning and development – Personal budgets for books and training courses to help you grow in your role. Plus 2 days a year - on us! - to further your skillset 🤓

• Private health care - Enjoy all the benefits Vitality has to offer, including reduced gym memberships and discounts on smartwatches 🏥

• Medical cash plan - To help you with the costs of dental, optical and physio (plus more!)

• Tech scheme - Get the latest tech for less 🖥

Plus all the rest; 33 days holiday, pension, cycle to work scheme, monthly team socials and company-wide socials every month!

Our process

We break it up into a few stages:

• Initial call with our Talent Acquisition Partner - 30 mins

• A past experience interview where you will discuss your journey so far and ways of working with Paul, our Director of Data Science & AI - 60 mins

• A technical interview with one of our Principal Data Scientists + Sr. ML Ops Engineers - 60 mins

• A culture interview with a bar raiser to see if your work style fits our processes and values (and vice versa!) - 60 mins

Everyone belongs at Marshmallow

At Marshmallow, we want to hire people from all walks of life with the passion and skills needed to help us achieve our company mission. To do that, we're committed to hiring without judgement, prejudice or bias.

We encourage everyone to apply for our open roles. Gender identity, race, ethnicity, sexual orientation, age or background does not affect how we process job applications.

We're working hard to build an inclusive culture that empowers our people to do their best work, have fun and feel that they belong.

Recruitment privacy policy

We take privacy seriously here at Marshmallow. Our Recruitment privacy notice explains how we process and handle your personal data. To find out more please view it here.