ML Systems Engineer, Large-Scale Model Training & RL Infrastructure
Palo Alto, California, United States
USD 195K-262K Senior-level Full Time
Tasks
- Build RL rollout and reward serving components
- Build distributed training infrastructure
- Build replay data buffer and checkpointing
- Create reproducible training runs and tooling
- Diagnose distributed training failures
- Implement parallelism strategies
- Integrate model training frameworks
- Partner with research scientists to deploy training recipes
- Profile and optimize GPU utilization
- Write design and incident documentation
Perks/Benefits
- 401k plan
- Career growth and learning opportunities
- Collaborative culture
- Disability insurance
- Flexibility and ownership
- Health insurance
- Life insurance
- Parental leave
- Remote Work Reimbursement
Skills/Tech-stack
CUDA | Checkpointing | DTensor | Data parallelism | DeepSpeed | Expert parallelism | FSDP | GPU Profiling | Infiniband | Kubernetes | Megatron-LM | NCCL | Pipeline parallelism | PyTorch | Python | RDMA | Ray | RoCE | Slurm | Tensor Parallelism | Transformer training | Triton
Education
N/A
Regions
Countries
States
Cities
Related jobs
-
Featured Feat. Principal Knowledge & Data Architect USD 174K-284KAWS Neptune | Canonicalization | Chunking | Cypher | DBTBenefits including health and wellness programs | Health, wellness, and retirement plansSenior-level Full TimeHeadquarters - Chevy Chase, MD R13d ago
-
Senior Data Engineer USD 170K-200KApache Airflow | Apache Spark | ClickHouse | Data Contracts | Data Modeling401k | Catered lunches | Commuter benefits | Continued education benefit | Equity compensationSenior-level Full TimeNew York, New York, United States6h ago
-
Senior Data Engineer USD 170K-200KApache Airflow | Apache Spark | Data Contracts | Data Modeling | Data Quality401k | Catered lunches | Commuter benefits | Company wellness day off | Continued education benefitSenior-level Full TimeLos Angeles, California, United States6h ago
-
Senior Data Engineer USD 170K-200KApache Airflow | Apache Spark | Data Contracts | Data Modeling | Data Quality401k | Catered lunches | Commuter benefits | Continued education benefit | FSASenior-level Full TimeSan Francisco, California, United States6h ago
-
AI Engineer USD 113K-180KAngular | Apache Kafka | CI/CD | Context Management | DevOps401k matching | Charitable matching | Community outreach groups | Flexible paid time off | Life insuranceMid-level Full TimeLeawood, KS, US R7h ago
-
Applied AI Engineer USD 80K-120KAPIs | Agent Orchestration | Data Pipelines | Evaluation | FedRAMPDental insurance | Equity compensation | Health insurance | Travel benefits | Vision insuranceMid-level Full TimeSF Bay Area, CA, USA7h ago
-
AWQ | Benchmarking | CUDA | CUDA kernels | Chunked prefill401k match | Career growth | Collaborative culture | Disability insurance | FlexibilitySenior-level Full TimePalo Alto, California, United States13h ago
-
APIs | Behavior Trees | C++ | Dashboards | Data PipelinesSenior-level Full TimeIrvine, CA13h ago
-
AI Engineer USD 230K-315KAPI Design | Caching | Concurrent programming | Context Window Pruning | Context caching401k matching | Comprehensive medical/dental/vision coverage | Fertility benefits | Flexible PTO | Lunch CreditSenior-level Full TimeSan Francisco, CA13h ago
-
Senior-level Full TimeColumbia, MD14h ago
-
Software Engineer - Platform & Application USD 175K-315KAPI serving | AWS | Agent Frameworks | Apache Airflow | Apache FlinkCollaborative team environment | Mentorship | Production responsibilityMid-level Full TimeSanta Clara HQ14h ago
-
C++ | Computer Vision | Data pipeline | Deep learning | EvaluationSenior-level Full TimeMountain View, California (HQ)14h ago
-
Senior Embedded Software Engineer - Future Forward USD 134K-201KAuthentication | C# | C++ | CAN | CUDASenior-level Full TimeSunnyvale, CA, United States R14h ago
-
Applied AI Lead USD 130K-150KAI Agents | AWS | AWS Bedrock | Anthropic Claude | Azure401k match | Dental insurance | EAP | Life and ADandD insurance | Medical insuranceSenior-level Full TimeHouston, TX, United States14h ago
-
Power Platform Analytics Engineer USD 100K-203KAI Builder | AI Search | Azure AI | Azure AI Search | Azure DataSenior-level Full TimeWashington, DC14h ago
-
Batch ML serving | DPO | Deep learning | Fine Tuning | InferenceSenior-level Full TimeUS-CA-Menlo Park14h ago
-
Research Operations, Code USD 130K-250KData pipeline | Experiment design | LLM | Language Processing | Machine LearningDental insurance | Equity grant | Free fitness membership | Health insurance | Housing bonusSenior-level Full TimeSan Francisco R15h ago
-
Senior Data Engineer USD 401KAzure Data | Azure Data Factory | Azure Synapse | CI/CD | Data Factory401k company match | Employee life insurance | Health insurance | Inclusive leave | Personal Time OffSenior-level Full TimeErlanger, KY15h ago
-
Senior Data Engineer - Modern Data & AI Platforms USD 156K-204KAWS | DBT | Data Governance | Data Ingestion | Data ModelingRemote workSenior-level Full TimeKansas City, MO, United States15h ago
-
Senior SoC Power Analysis & Optimization Engineer USD 120K-600KAI workload | AI workload characterization | ASIC | Clock gating | Gate Level Analysis401k match | Company holidays | Flexible spending accounts | Health, dental, vision, and life insurance | Home tech setup reimbursementSenior-level Full TimeMountain View, CA R15h ago
-
Production AI Engineer USD 145K-165KAWS | AWS CloudFormation | Automation | CI/CD | Cloud Orchestration401k plan | Basic life insurance | Company match | Employee stock purchasing plan | Healthcare benefitsMid-level Full TimeAtlanta, GA, United States16h ago
-
Systems Engineer USD 120K-197KAgile | Amazon ECS | Amazon EKS | Amazon Web Services | Bash100 percent paid healthcare and dental insurance | 401k matching | Escalating vacation time | Paid parental leave | Professional development trainingSenior-level Full TimeGaithersburg, MD16h ago
-
Field Engineer, Public Sector USD 180K-226K2D Image Annotation | API Design | API Integrations | Ansible | CI/CDCommuter stipend | Dental insurance | Health insurance | Learning & development stipend | Paid time offSenior-level Full TimeAustin, TX; St. Louis, MO; Washington, …16h ago
-
Data Analyst, Financial Data Engineering USD 161K-242KAirflow | Alerting | Apache Spark | Code review | Data Modeling401k plan | Company bonus | Equity | Medical, dental, vision benefits | Wellness stipendsSenior-level Full TimeNew York18h ago
-
Senior AI Engineer (m/f/d) USD 100K-140K.NET | AI Search | AIOps | API Integration | AWS100 percent salary during sick leave | 4 weeks remote work per year | Childcare support | Coursera access | Experience daysSenior-level Full TimeMadrid, MD, Spain R19h ago