Lead ML Ops Engineer

About Marshmallow

We build financial products that accelerate the economic freedom for people who move across borders. We started with car insurance — insuring over a million drivers — and we’re scaling beyond. Tens of millions of people move countries each year, facing overlooked financial challenges. Our future is in building financial products around their needs to positively impact their lives.

How we work

We’re really proud of the culture we’ve created. We push for progress every day, because we know that we’ll only hit big milestones by taking lots of smaller steps. We’re always open to helping our team mates, sharing our ideas, experience and knowledge to solve problems together. We take risks, think creatively and experiment relentlessly to meet our customer’s needs, and never pass blame when things go wrong. We encourage people at all levels to take ownership of their work, and to be bold in challenging how we do things. Everyone has a voice and the opportunity to make an impact.
And autonomy and ownership are only possible with clear direction. That’s why we collaborate to do in-depth planning twice a year, and make sure we leave with clear goals and objectives that flow from top to bottom. To make sure we’re as aligned as possible across functions, most of our work rolls up into three tribes; Acquisition, Retention & Claims. Each tribe has multiple teams embedded in it, working cross-functionally to do great work.
We’re so excited for all of the challenges up ahead, and we need more people to help us tackle them! If life at Marshmallow sounds like it could be for you, explore our Culture Handbook to find out more.

The Data Science Team

Our central Data Science & AI team owns the end‑to‑end lifecycle of every model at Marshmallow, from experimentation to production. As an insurer‑fintech, serving hundreds of thousands of customers, reliability and speed are critical - our models quote prices, detect fraud and power next‑gen GenAI experiences in real time.

You will lead the ML Ops function within the team, line‑managing two ML Ops Engineers and reporting to the Director of Data Science & AI. Day‑to‑day you will partner with pricing and central data scientists, platform engineers and external vendors to ensure our models ship quickly, safely and at scale.

What you’ll be doing

• Crafting and owning a multi‑year ML Ops roadmap that unlocks <24 h model iteration cycles and 99.9 %+ production uptime.

• Scanning the horizon for emerging ML and GenAI technologies, and translating insights into practical pilots that keep Marshmallow at the forefront of innovation.

• Setting and iterating the company‑wide MLOps & LLMOps strategy, covering classical ML and emerging generative‑AI workloads.

• Hiring, coaching and inspiring a high‑performing ML Ops Engineering team with a culture of ownership and excellence.

• Mentoring and developing the ML Ops Engineers, expanding the team as we grow and setting technical standards while promoting best practice across Data Science and Engineering.

• Acting as the go‑to person for ML Ops questions from senior stakeholders and championing robust ML deployment across the business.

• Making build‑versus‑buy decisions, evaluating new vendors and tooling, and negotiating contracts where relevant.

• Championing a “You build it, you run it” culture so every data scientist can safely own their models in production.

• Overseeing multiple deployment pipelines on AWS SageMaker (real‑time and batch inference) and integrating them with TeamCity for automated CI/CD.

• Managing the relationship with Tecton as our feature platform vendor, advising Data Science teams on its use for training and inference, and overseeing related infrastructure cost and operation

Who you are

You are a strategic engineering leader who blends long‑term thinking with hands‑on delivery. You can articulate a compelling 3 to 5-year vision for ML Ops at a scaling fintech, then translate that into an actionable roadmap. You thrive in fast‑growing environments, where ownership is key and ambiguity is the norm. You continually scan the horizon for emerging ML and GenAI technologies and rapidly assess how they should reshape our tool‑chain and operating model. You have a track record of defining and owning ML Ops strategy, then landing it successfully to unlock measurable commercial value. You build high‑performing teams through clear direction, coaching and a strong engineering culture, and you hold a high bar for reliability, observability and developer experience.

What we're looking for from you

Experiences that are essential

• End-to-end ownership of ML platforms servicing real-time, customer-facing products in regulated industries (finance, insurance, etc.).

• Demonstrable success in hiring, developing and retaining high‑performing engineering teams, with evidence of managing, mentoring or leading.

• Proven ownership of an ML Ops or ML platform roadmap that delivered clear business impact.

• Production experience with the AWS ecosystem, especially SageMaker for training and hosting.

• Strong Python skills and the ability to work and integrate with Java backend teams.

• Hands‑on delivery of CI/CD for ML (e.g. TeamCity, Jenkins, GitHub Actions).

• Infrastructure‑as‑code with Terraform (or similar) and containerisation with Docker.

• Designing monitoring solutions that detect performance, drift and data‑quality issues in production.

• Leading or mentoring engineers, and influencing cross‑functional stakeholders.

Experiences that will help you

• Working with feature stores such as Tecton.

• Implementing or integrating model‑monitoring platforms like Arize.

• Deploying or operating LLM‑based applications at scale.

• Negotiating with vendors and managing third‑party contracts.

• Familiarity with Kubernetes or serverless architectures on AWS.

• An understanding of FCA compliance considerations for ML services.

• Comfortable with vector DBS, embedding management and model lifecycle tooling

• Experience with model parallelism, distributed training

Perks of the job

• Flexi-office working – Spend 2-3 days a week with your team in our new collaborative London office. The rest is up to you! 🏠

• Competitive bonus scheme - designed to reward and recognise high performance 🌟

• Flexible benefits budget - £50 per month to spend on a Ben Mastercard, meaning you get your own benefits budget to spend on things you want. Whether that’s subscriptions, night classes (puppy yoga, anyone?), the big shop or a forest of houseplants. Pretty much anything goes 💰

• Sabbatical Leave - Get a 4-week fully paid sabbatical after being with us for 4 years 🏝️

• Work From Anywhere - 4 weeks work from anywhere to use, with no need to come to the office 🛫

• Mental wellbeing support – Access therapy and mental health sessions through Oliva 💚

• Learning and development – Personal budgets for books and training courses to help you grow in your role. Plus 2 days a year - on us! - to further your skillset 🤓

• Private health care - Enjoy all the benefits Vitality has to offer, including reduced gym memberships and discounts on smartwatches 🏥

• Medical cash plan - To help you with the costs of dental, optical and physio (plus more!)

• Tech scheme - Get the latest tech for less 🖥

Plus all the rest; 33 days holiday, pension, cycle to work scheme, monthly team socials and company-wide socials every month!

Our process

We break it up into a few stages:

• Initial call with our Talent Acquisition Partner - 30 mins

• A past experience interview where you will discuss your journey so far and ways of working with Paul, our Director of Data Science & AI - 60 mins

• A technical interview with one of our Principal Data Scientists + Sr. ML Ops Engineers - 60 mins

• A culture interview with a bar raiser to see if your work style fits our processes and values (and vice versa!) - 60 mins

Everyone belongs at Marshmallow

At Marshmallow, we want to hire people from all walks of life with the passion and skills needed to help us achieve our company mission. To do that, we're committed to hiring without judgement, prejudice or bias.

We encourage everyone to apply for our open roles. Gender identity, race, ethnicity, sexual orientation, age or background does not affect how we process job applications.

We're working hard to build an inclusive culture that empowers our people to do their best work, have fun and feel that they belong.

Recruitment privacy policy

We take privacy seriously here at Marshmallow. Our Recruitment privacy notice explains how we process and handle your personal data. To find out more please view it here.