Software Technical Leader - GenAI Gateway

Tasks

• Apply model quantization
• Design serving architecture
• Develop gateway core components in Go
• Enable horizontal scalability
• Ensure high availability
• Implement prompt caching
• Implement token compression
• Lead model inference strategy
• Mentor engineers
• Optimize cost and latency
• Optimize kv cache
• Review technical designs

Perks/Benefits

• Hybrid work

Skills/Tech-stack

CUDA | Concurrency | GPU | GRPC | Go | KV cache | LLM | Memory Management | Model Optimization | Open Source | Open-source Models | Prompt Caching | Quantization | Token Compression