AI-Factory Observability Principal
Tasks
- Correlate network behavior with training performance
- Define SLOs alerting strategy and on call signal quality
- Define observability architecture
- Design unified data model for correlation
- Establish instrumentation and telemetry pipelines
- Generate incident summaries with LLMs
- Implement AIOps anomaly detection and forecasting
- Instrument Kubernetes for AI workloads
- Integrate BMS telemetry into observability platform
- Integrate EPMS power telemetry into observability platform
- Mentor engineers and set observability roadmap direction
- Monitor GPU accelerator clusters
- Own metrics logs traces and events
Perks/Benefits
- N/A
Skills/Tech-stack
AIOps | Anomaly Detection | BACnet | BMS | DNP3 | Datacenter Engineering | Datadog | Dynatrace | ECC errors | EPMS | Elasticsearch | Events | Forecasting | GNMI | GPU Monitoring | Grafana | IEC 61850 | IPMI | Infiniband | Infrastructure as Code | Kubernetes | Kubernetes Scheduler | LLM | Language Processing | Linux | Logs | Loki | Machine Learning | Metrics | Modbus | Mqtt | NVIDIA DCGM | NVLink | Natural Language | Natural Language Processing | Netflow | OPC UA | Observability | OpenTelemetry | Openconfig | Predictive Maintenance | Prometheus | Python | RDMA | Redfish | RoCE | SFlow | SNMP | SRE | Splunk | Streaming telemetry | Telemetry pipelines | Tempo | Terraform | Thanos | Time Series | Time-series databases | Traces | VictoriaMetrics | WMI | “as-code”
Education
N/A
Related jobs
-
(US) Principal ML System Engineer USD 139K-304KAWS | Access Control | Alerting | Audit Logging | AuthenticationSenior-level Full TimeRemote, USA R8h ago
-
Senior Machine Learning Engineer, Ads ML Platform USD 216K-303KAnomaly Detection | Data Governance | Data Infrastructure | Data Lineage | Data Storage401k employer match | Caregiving support | Comprehensive healthcare benefits | Family planning support | Flexible remote workSenior-level Full TimeRemote - United States R11h ago
-
Senior Software Engineer - Storage Control Plane USD 266K-395KBlock Storage | C# | C++ | CI/CD | CephSenior-level Full TimeSan Francisco Office (Fremont St) R12h ago
-
A/B | A/B Testing | B testing | Data parallelism | Deep learning401k employer match | Family planning support | Flexible vacation | Gender-affirming care | Global benefit programsSenior-level Full TimeRemote - United States R13h ago
-
A/B | A/B Testing | B testing | C++ | Cloud Computing401k employer match | Family planning support | Flexible vacation | Gender-affirming care | Global benefitsSenior-level Full TimeRemote - United States R13h ago
-
Applied AI/ML Engineer USD 150K-200KAPI Integration | Anomaly Detection | Apache Spark | CI/CD | Cloud Platforms401k match | Comprehensive benefits package | Flexible PTO | Paid Holidays | Remote workMid-level Full TimeWashington DC area R14h ago
-
Finance & Strategy AI Engineer USD 146K-246KAPI Integration | Adaptive Planning | Agentic Workflows | BigQuery | Data GovernanceFlexible remote work | Health benefits | Parental leave | Professional development stipendSenior-level Full TimeSan Francisco - SF9 R14h ago
-
Senior DevOps Engineer (Storage) - remote in the US USD 200K-290KAir-gapped | ArgoCD | CAPI | CSI | Cluster APICompany outings | Conference attendance | Hackathons | Happy hours | Professional development and trainingExecutive-level Full TimeRemote, USA, United States R15h ago
-
Senior Software Engineer (Storage) - remote in the US USD 100K-170KArgoCD | CSI | Cluster API | Flux | GitOpsCompany outings and happy hours | Conferences and working groups | Hackathons and tech talks | Professional development and trainingSenior-level Full TimeRemote, USA, United States R15h ago
-
Autonomy Algorithms Software Engineer USD 115K-145KAgent systems | Agile | Algorithms | C++ | CI/CDHybrid work environment | Remote work optionMid-level Full TimeArlington, VA R15h ago
-
Autonomy Algorithms Software Engineer USD 115K-145KAgent systems | Agile | Algorithms | C++ | CI/CDFlexible work schedule | Hybrid work environment | Remote work availableMid-level Full TimeWoburn, MA R15h ago
-
Forward Deployed Engineer USD 280K-320KAgent Development | Deployment | Evaluation | JavaScript | Language ModelsFlexible working hours | Generous vacation | Parental leave | Visa sponsorshipMid-level Full TimeNew York City, NY; San Francisco, … R16h ago
-
Software Engineer, Embedded Agentic AI USD 170K-210KC# | C++ | Cloud Platforms | Containerization | DockerDisability benefits | Equity awards | Health insurance | Life insurance | Paid time offEntry-level Full TimeSan Jose, California R16h ago
-
Senior Software Engineer, Data Platform USD 196K-245KAirflow | BigQuery | Cost Optimization | Dagster | Data GovernanceBenefits | Equity | Relocation assistanceSenior-level Full TimeSan Francisco Bay Area or Remote … R16h ago
-
Senior Research Engineer, LLM Training & Post-Training USD 165K-310KCUDA | DPO | DeepSpeed | Distributed Training | Experiment design401k matching | Company holidays | Comprehensive health coverage | Dental coverage | Flexible time offSenior-level Full TimeNew York, New York, United States; … R16h ago
-
Robotics Test Engineer USD 140K-170KAQL | Accelerated life testing | Build Management | CAD | Cause analysis100 percent employer paid healthcare | 401k employer matching | Annual professional development stipend | Complimentary dinner while working late | Complimentary recovery sessions for self friends and familySenior-level Full TimeNYC/Metro Area R17h ago
-
Senior Software Development Engineer - Iris AI USD 130K-162K.NET | API Design | Amazon Bedrock | Anthropic API | Asynchronous ArchitectureEmployee assistance program | Health insurance | Life insurance | Paid Company Holidays | Paid time offSenior-level Full TimeU.S. Remote R18h ago
-
Specialist Solutions Architect - Data Engineering & Warehousing (Digital Native Business) USD 180K-247KAWS | Anomaly Detection | Apache Spark | Cloud platform | Data ModelingContinuous learning | Mentorship | Remote work | Travel up to 30 percentSenior-level Full TimeUnited States R18h ago
-
Enterprise Applied AI Solution Specialist USD 90K-130KAPI Integration | Agent Frameworks | Conversational Design | Data Governance | Function CallingMid-level Full TimeWeston,FL,United States R18h ago
-
Senior AI Data Engineer USD 118K-178KAI Agent | BigQuery | CRM | Customer Data | Customer Data PlatformFlexible working model | Health insurance | Parental leave | Professional development stipend | Remote workSenior-level Full TimeRemote - US R18h ago
-
AWS | AWS EKS | Azure | Azure AKS | CI/CD401k match | ESPP | Family leave programs | Flexible time off | Flexible-hybrid workSenior-level Full TimeSan Diego, CALIFORNIA, United States R18h ago
-
AWS | Azure | CI/CD | Cloud platform | Containers401k matching | Company matching donations | ESPP | Family leave programs | Flexible hybrid work scheduleSenior-level Full TimeSan Diego, CALIFORNIA, United States R18h ago
-
Machine Learning Data Scientist, Forecasting USD 230K-385KBayesian Methods | Causal Inference | Experimentation | Machine Learning | Model MonitoringHybrid work model | Relocation assistanceSenior-level Full TimeSan Francisco R19h ago
-
Data Engineer I - Salesforce Data 360 USD 87K-123KAPIs | Access Controls | Data Governance | Data Lineage | Data Modeling401k match | Career growth and development programs | Comprehensive onboarding | Conference participation | Employee resource groupsMid-level Full TimeMorrisville, North Carolina (NC), United States R19h ago
-
Lead Business Operations Data Engineer - 928 USD 158K-200KAirflow | Alerting | Azure Data | Azure Data Factory | DBTHybrid work | Remote work up to 2 days per week with approvalSenior-level Full TimeUS Broomfield, CO R19h ago