LLM Ops Engineer
Tasks
- Abstract prompts from application code
- Build and maintain GenAI middleware
- Conduct trace analysis for optimization
- Create and maintain golden datasets
- Create dashboards for model performance
- Create prompt templates
- Design user feedback collection systems
- Detect and mitigate bias in model outputs
- Develop and manage model serving APIs
- Develop output validation frameworks
- Ensure regulatory compliance for AI systems
- Fine-tune pretrained models
- Implement content moderation systems
- Implement continuous improvement processes
- Implement model optimization techniques
- Implement statistical validation for model outputs
- Integrate LLMs with existing systems and data sources
- Manage LLM lifecycle from development to deployment
- Manage training infrastructure and compute environments
- Monitor hallucination rates and accuracy
- Monitor latency and throughput metrics
- Optimize inference and training costs
- Prepare training datasets
- Run A B tests for models and features
- Scale inference infrastructure for demand loads
- Set confidence score thresholds for production
- Set up prompt testing environments
- Use similarity matching for response evaluation
- Version and store prompts
Perks/Benefits
- 401k company match
- Dental insurance
- Health insurance
- Incentive programs
- Paid time off
- Recognition programs
- Retirement savings plan
- Vision insurance
- Work-life balance
Skills/Tech-stack
A/B | A/B Testing | AI Safety | API Development | Amazon SageMaker | Apache Airflow | Azure OpenAI | Azure OpenAI Service | B testing | Bias detection | Content Moderation | Cost Optimization | Data Validation | Fine Tuning | Google Vertex | Google Vertex AI | Infrastructure Scaling | Kubeflow | LLM Fine-tuning | Latency optimization | Learning operations | MLOps | MLflow | Machine Learning | Machine Learning Operations | Model Serving | Monitoring Dashboards | OpenAI Service | Prompt engineering | Python | Vertex AI
Education
N/A
Related jobs
-
Senior-level Full TimeAustin, TX, US7h ago
-
Mid-level Full TimeUnited States7h ago
-
Applied AI Engineer - Life Sciences & Healthcare USD 150K-200KAWS | Amazon S3 | CI/CD | Caching | Claims ProcessingSenior-level Full TimeAustin, TX, US8h ago
-
AWS | Amazon S3 | Artificial Intelligence | Cloud Databases | Data AnalysisEntry-level ContractNew York, United States8h ago
-
Active Learning | Bayesian Inference | Bayesian optimization | Cheminformatics | Machine Learning401k | Dental insurance | Medical insurance | Paid sick leave | Vision insuranceMid-level ContractSouth San Francisco, United States9h ago
-
AI Governance | AI guardrails | API Integration | Agent workflows | AutoGPTSenior-level ContractAustin, United States9h ago
-
Staff Engineer – AI/ML & Digital Twin USD 112K-168KAWS | Autoencoders | Azure | Cloud Computing | Convolutional Neural NetworksSenior-level Full TimePennsylvania, Canonsburg9h ago
-
Machine Learning Engineer Intern - Global E-Commerce Content Recommendation - 2026 Summer (BS/MS) USD 115K-168KData Engineering | Diffusion Models | Fairness | Generative Models | Language ModelsHands-on experience | Industry exposure | Social events | WorkshopsEntry-level InternshipSan Jose, California, United States9h ago
-
Machine Learning Engineer, Search - Local Services Team USD 194K-355KBlended Ranking | CTR Prediction | CVR Prediction | Information Retrieval | Language ProcessingEntry-level Full TimeSan Jose, California, United States9h ago
-
Computer Vision | Data Modeling | Language Models | Language Processing | Large Language ModelsEntry-level Full TimeSan Jose, California, United States9h ago
-
Research Engineer, MRS AI USD 117K-173KAgent Orchestration | Bias Mitigation | Continual Learning | Data Augmentation | DebiasingEntry-level Full TimeBellevue, WA | Menlo Park, CA …10h ago
-
Software Engineer, Play Analytics, Platforms and Devices USD 147K-211KArtificial Intelligence | C++ | Code Quality | Computer networks | Data StorageMid-level Full TimeMountain View, CA, USA10h ago
-
API Design | Asynchronous programming | Backend Development | Computer Vision | Data ProcessingSenior-level Full TimeSunnyvale, CA, USA10h ago
-
Software Engineer III, Machine Learning, Android AI USD 147K-211KAI Agents | Computer Vision | Data Pipelines | Data Processing | Data cleaningSenior-level Full TimeMountain View, CA, USA10h ago
-
Software Engineer III, Speech Production, Infrastructure USD 147K-211KAutomatic Speech Recognition | C++ | Data Structures | Data Structures and Algorithms | Distributed SystemsSenior-level Full TimeMountain View, CA, USA10h ago
-
Senior Software Engineer, Core Developer Intelligence USD 174K-252KArtificial Intelligence | Data Analysis | Data Pipelines | Data Storage | Distributed ComputingSenior-level Full TimeSunnyvale, CA, USA10h ago
-
Senior Software Engineer, AI/ML, Maps Navigation USD 174K-252KC plus plus | Code review | Computational Geometry | Computer Vision | Data StructuresSenior-level Full TimeMountain View, CA, USA10h ago
-
Software Engineer III, AI/ML GenAI, YouTube USD 147K-211KC++ | Computer Vision | Data Processing | Debugging | Language ModelsSenior-level Full TimeMountain View, CA, USA10h ago
-
Cloud platform | Computer Vision | Data Processing | Data Structures | Data Structures and AlgorithmsSenior-level Full TimeSunnyvale, CA, USA10h ago
-
Senior Software Engineer, AI/ML, AI and Infrastructure USD 174K-252KC++ | Data Processing | Data Storage | Debugging | Distributed ComputingSenior-level Full TimeSunnyvale, CA, USA10h ago
-
AI/ML Engineer, National Security, Google Public Sector USD 153K-222KC plus plus | Data Validation | Deep learning | Feature Engineering | GPUsTravel up to 30 percentSenior-level Full TimeReston, VA, USA; Washington D.C., DC, …10h ago
-
C++ | Clustering | Data Processing | Data Structures | Data Structures and AlgorithmsSenior-level Full TimeMountain View, CA, USA10h ago
-
Software Developer Backend and GenAI USD 147K-216KAlloyDB | BigQuery | CI/CD | Cloud Composer | Cloud RunSenior-level Full TimeUnited States19h ago
-
Data Scientist Lead - Payments USD 165K-205KAgile | Big Data | Distributed Computing | Language Processing | Machine LearningSenior-level Full TimeNew York, NY, United States19h ago
-
Senior Exploitation Specialist / Data Scientist USD 93K-166KData Analysis | Data Processing | Data Visualization | Machine Learning | PythonSenior-level Full TimeSpringfield, Virginia, United States20h ago