Backend Engineer – ML Systems AVP

Organization Overview:

Market Operations Technology is a dynamic, global and diverse Organization, supporting operations areas that have a presence in over 60 Countries.  Our Technology groups develop and implement a wide range of applications that support the Operations groups which are core to the success of the Markets business and has significant impact across the lifecycle of a Trade by providing innovative products and solutions. We partner extensively with a range of internal stakeholders including product aligned Middle Office groups, Settlements, Margin, Asset Servicing, Listed Derivative & Commodities Operations. We support an extensive range of Capital Markets products and services including Fixed Income (FX, Rates, Credit, Muni, Cash and Derivatives) where Citi is consistently a dominant top 3 player in the market, Equity (Cash, Derivatives, Prime Brokerage, Futures, Listed Derivatives, FXPB) where Citi has a significant growth program to continue to build out its capabilities and services; and Syndicates where we support both FI and EQ new issues and IPOs. We play a key role in supporting our global clients to ensure they have a first-class experience in service performance and strategic partnership.

ML Solutions team within Markets OPS Technology is dedicated to developing solutions using Artificial Intelligence, Machine Learning and Generative AI. This team is a leader in creating new ideas, innovative technology solutions, and ground-breaking solutions for Markets Operations and other Line of Businesses. We work closely with our clients and business partners to progress solutions from ideation to production by leveraging the entrepreneurial spirit and technological excellence.

Job Description:

ML Solutions team is seeking a skilled and innovative ML Systems Engineer with ML Ops experience to join our team. In this role, you will be responsible for building and maintaining scalable backend systems, deploying data science (DS) models, and designing robust pipelines that enable seamless integration of DS solutions into our applications. You will work closely with data scientists and front end engineers to help bridge the gap between data science and production-ready systems. If you're passionate about operationalizing data Science models, creating reliable backend services, and improving the lifecycle of data science workflows, this role could be a great fit for you.

Key Responsibilities:

• ML backend and Orchestration:
  • Hands on development of ML Systems backend infrastructure, messaging and integration with interfacing systems
  • Design, build, and maintain APIs, Microservices, and systems to serve data science models.
  • Ensure ML Infrastructure systems are scalable, reliable, and secure
• Database Management:
  • Work with SQLAlchemy to define, maintain, and query the database schema for an application.
  • Work with raw SQL to run advanced queries for things like alerting and reporting.
• ML Ops Implementation:
  • Develop and maintain CI/CD pipelines for data science models and services
  • Automate and track data science workflows with tools like MLflow
• Infrastructure:
  • Manage and optimize Kubernetes cluster via Openshift.
  • Implement and manage various infrastructure components such as PostgreSQL, Kafka, S3.
  • Knowledge of running workloads in AWS or GCP will be plus.
• Collaboration:
  • Work with data scientists to understand model requirements and ensure successful deployment into production systems
  • Collaborate with DevOps and infrastructure teams to improve scalability and reliability
• Performance & Optimization:
  • Optimize backend services and data science model inference for latency and throughput
  • Troubleshoot issues in production environments, ensuring high availability of services

Engineering Excellence & Best Practices:
 

• Drive the adoption of modern engineering ways of working, including Agile, DevOps, and CI/CD.
• Advocate for automated testing, infrastructure as code, and continuous monitoring to enhance software reliability.
• Apply Behavior-Driven Development (BDD), Test-Driven Development (TDD), and unit testing to ensure code quality and functionality.
• Conduct thorough code reviews, ensuring adherence to best practices in readability, performance, and security.
• Implement and enforce secure coding practices, performing vulnerability assessments and ensuring compliance with security standards.
• Collaborate effectively in agile environments, embracing DevOps principles and fostering a culture of continuous delivery and improvement.

Skills & Qualifications:

• Proficiency in Python
• Hands on experience working with a Python web framework such as FastAPI, Flask, Django, etc
• Peripheral knowledge in Machine Learning and Data Science
• Working knowledge in MLOps principals such as experiment tracking, model serving, model orchestration, etc.
• Hands on experience with designing DB driven applications and using an ORM such as SQLAlchemy.
• Hands on experience with deploying and maintaining production applications
• Hands on experience with managing and debugging the necessary infrastructure to support a full stack application
• Hands on experience with Kubernetes
• Hands on experience working with stream processing tools like Kafka or Apache Spark
• Bachelor’s degree or equivalent experience in Computer Science or a related field.

Ideal skills:

• Hands on experience working with a Python orchestrator (Dagster, Airflow, Prefect, etc)
• Hands on experience working with various MLOps tools such as MLFlow, Kedro, etc
• Hands on experience working with LLMs and related technologies such as Vector DBs, Agents, etc.

------------------------------------------------------

Job Family Group:

------------------------------------------------------

Job Family:

------------------------------------------------------

Time Type:

------------------------------------------------------

Citi is an equal opportunity and affirmative action employer.

Qualified applicants will receive consideration without regard to their race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or status as a protected veteran.

Citigroup Inc. and its subsidiaries ("Citi”) invite all qualified interested applicants to apply for career opportunities. If you are a person with a disability and need a reasonable accommodation to use our search tools and/or apply for a career opportunity review Accessibility at Citi.

View the "EEO is the Law" poster. View the EEO is the Law Supplement.

View the EEO Policy Statement.

View the Pay Transparency Posting