Senior LLMOps Engineer

Are you ready to power the World's connections?
About The Role
Kong’s API Management suite including the Kong Enterprise AI Gateway and Konnect support multiple AI capabilities for customers. The AI Gateway offers a unified API for managing multiple Large Language Models (LLMs), allowing users to switch between different AI providers seamlessly. The AI Gateway also allows a variety of AI policies to be enforced such as semantic caching, advanced routing and load balancing, promt guards and Personally Identifiable Information (PII) sanitization. The Konnect SaaS platform exposes these same AI capabilities via Cloud Gateways and provides customers a variety of AI-driven insights into the operations of a customer’s API surface.We are building out a new platform team - LLMOps - in our Bangalore office to help all the other engineering teams in Kong work with Large Language Models (LLMs) and build new features into our product portfolio.
What You'll Do
Be the founding member of the LLMOps teamProvide an LLM SaaS platform for all the engineering teams at Kong, giving them access to a variety of LLMs based on their needsManage model lifecycle with high reliability and uptime SLOs (at least 99.99%)Fine tune the platform to provide AI security and high throughput for LLM queriesOwn the end-to-end deployment architecture of the LLM SaaS platformPlay a pivotal role in shaping the technical direction for AI innovation in KongCollaborate closely with product leadership to define and refine strategy, roadmap, and objectives for the LLM SaaS platforms, ensuring alignment between engineering efforts and business goals.Assist in the hiring process and help grow and mentor the team
What You'll Bring
Bachelor's or Master's degree in Computer Science or a related field5+ years of experience in building and operating highly reliable SaaS/PaaS systemsPrior ownership of uptime SLOs for a platform operating at 99.99% or higherHands-on experience with at least one of the major cloud platforms (AWS, Azure, or GCP)Strong experience with AI platforms in public clouds (like Bedrock, Vertex AI, etc.).Familiarity with observability tools such as Datadog, Prometheus, Grafana, Victoria Metrics, Loki, or similar technologies.Expertise in designing and developing highly scalable distributed systemsExperience managing incidents and communicating effectively under high-pressure situationsStrong verbal and written communication skills to effectively collaborate across teams