AI-Factory Observability Principal
Tasks
- Correlate network behavior with training performance
- Define SLOs alerting strategy and on call signal quality
- Define observability architecture
- Design unified data model for correlation
- Establish instrumentation and telemetry pipelines
- Generate incident summaries with LLMs
- Implement AIOps anomaly detection and forecasting
- Instrument Kubernetes for AI workloads
- Integrate BMS telemetry into observability platform
- Integrate EPMS power telemetry into observability platform
- Mentor engineers and set observability roadmap direction
- Monitor GPU accelerator clusters
- Own metrics logs traces and events
Perks/Benefits
- N/A
Skills/Tech-stack
AIOps | Anomaly Detection | BACnet | BMS | DNP3 | Datacenter Engineering | Datadog | Dynatrace | ECC errors | EPMS | Elasticsearch | Events | Forecasting | GNMI | GPU Monitoring | Grafana | IEC 61850 | IPMI | Infiniband | Infrastructure as Code | Kubernetes | Kubernetes Scheduler | LLM | Language Processing | Linux | Logs | Loki | Machine Learning | Metrics | Modbus | Mqtt | NVIDIA DCGM | NVLink | Natural Language | Natural Language Processing | Netflow | OPC UA | Observability | OpenTelemetry | Openconfig | Predictive Maintenance | Prometheus | Python | RDMA | Redfish | RoCE | SFlow | SNMP | SRE | Splunk | Streaming telemetry | Telemetry pipelines | Tempo | Terraform | Thanos | Time Series | Time-series databases | Traces | VictoriaMetrics | WMI | “as-code”
Education
N/A
Related jobs
-
Featured Feat. Principal Knowledge & Data Architect USD 174K-284KAWS Neptune | Canonicalization | Chunking | Cypher | DBTBenefits including health and wellness programs | Health, wellness, and retirement plansSenior-level Full TimeHeadquarters - Chevy Chase, MD R14d ago
-
Featured Feat. Associate Director, Data Labs USD 167K-167KAWS | Cloud Computing | Compute Infrastructure | Data Analysis | LLM GovernanceConference speaking opportunities | Hybrid work schedule | Media appearancesSenior-level Full TimeWashington, District of Columbia, 20004, United … R27d ago
-
AI Engineer USD 150K-210KEvaluation systems | LLM APIs | Language Models | Large Language Models | PostgreSQL401k plan | Flexible time off | Flexible work schedules | Health and wellness benefits | In-person offsitesMid-level Full TimeSan Francisco, CA / Remote (US) R6h ago
-
Senior DevOps Engineer (Storage) - remote, US/EU USD 202K-290KAir-gapped | Air-gapped environments | Argo CD | ArgoCD | BashConference attendance | Hackathons | Professional development and training | Remote work | Tech talksExecutive-level Full TimeAustin, TX, United States R13h ago
-
AI Engineer USD 113K-180KAngular | Apache Kafka | CI/CD | Context Management | DevOps401k matching | Charitable matching | Community outreach groups | Flexible paid time off | Life insuranceMid-level Full TimeLeawood, KS, US R15h ago
-
Senior Data Science Engineer USD 116K-198KApache Airflow | CI/CD | DBT | Databricks | Databricks AutoloaderFlexible work optionsSenior-level Full TimeRemote (United States) R21h ago
-
Senior Embedded Software Engineer - Future Forward USD 134K-201KAuthentication | C# | C++ | CAN | CUDASenior-level Full TimeSunnyvale, CA, United States R23h ago
-
Research Operations, Code USD 130K-250KData pipeline | Experiment design | LLM | Language Processing | Machine LearningDental insurance | Equity grant | Free fitness membership | Health insurance | Housing bonusSenior-level Full TimeSan Francisco R23h ago
-
Senior SoC Power Analysis & Optimization Engineer USD 120K-600KAI workload | AI workload characterization | ASIC | Clock gating | Gate Level Analysis401k match | Company holidays | Flexible spending accounts | Health, dental, vision, and life insurance | Home tech setup reimbursementSenior-level Full TimeMountain View, CA R1d ago
-
Azure Data Engineer USD 110K-140K3NF | API Management | Agile Scrum | Azure Data | Azure Data Factory401k match | Employee resource group network | Holiday pay | Long-term disability | Medical, dental & vision coverageSenior-level Full TimeUnited States R1d ago
-
Production Support Senior Analyst USD 113K-151KAPI Integration | APIs | Agile | Backlog Management | Dashboard DevelopmentHealth and life insurance | Hybrid work model | Paid time off | Personal and family care leaves | Retirement benefitsSenior-level Full TimeJersey City, NJ, United States R1d ago
-
Lead AI Engineer USD 130K-201KAI Search | App Service | Azure AI | Azure AI Search | Azure App401k match | Employee Resource Group | Health insurance | Paid time off | Parental leaveSenior-level Full TimeUnited States R1d ago
-
Senior AI Engineer (m/f/d) USD 100K-140K.NET | AI Search | AIOps | API Integration | AWS100 percent salary during sick leave | 4 weeks remote work per year | Childcare support | Coursera access | Experience daysSenior-level Full TimeMadrid, MD, Spain R1d ago
-
Sr. Forward Deployed Engineer-Retail USD 182K-250KAI APIs | AWS | Apache Spark | Artificial Intelligence | AzureCustomer site travelSenior-level Full TimeRemote - California R1d ago
-
Data Engineer USD 105K-140KAPIs | AWS | Apache Airflow | Apache Iceberg | Apache Kafka401k plan with company match | Annual bonus | Company equipment provided | Company events and team outings | Company medical dental and vision plansMid-level Full TimeAtlanta, GA preferred, Remote R1d ago
-
Behavioral Data | Data Pipelines | Data Quality | Distributed Systems | Evaluation metricsGlobal annual leave policy | In-person onboarding | Remote work | Restricted stock units (RSUs)Senior-level Full TimeUnited States (Remote) R1d ago
-
AI Governance | AI platforms | API Integration | Agentic Systems | ComplianceEnglish classes | Paid time off | Performance coaching | Remote work | Training and developmentMid-level Full TimeRemote R1d ago
-
AI Forward Deployed Engineer | $120K-$150K + Hybrid + Equity | AI-Based Outage Intelligence Company A USD 120K-150KAI workflows | API Integration | Automation | Backend Development | Customer IntegrationDirect impact on company growth | Equity | Fast-moving startup environment | Hybrid work environment | Significant ownershipMid-level Full TimeKing of Prussia, PA, United States R1d ago
-
Senior Data Scientist, Creator Platform USD 263K-322KAI Evaluation | Agentic AI | Anti Abuse Modeling | Anti-abuse | Causal InferenceEquity compensation | Health insurance | Onsite work scheduleSenior-level Full TimeSan Mateo, CA, United States R1d ago
-
Computational Biology MLOps Engineer USD 125K-189KAWS | Alerting | CI/CD | Checkpointing | Cloud platformOnsite work three days per weekMid-level Full TimeSan Diego, CA / Indianapolis, IN … R1d ago
-
Software Engineer, AI Developer Experience USD 171K-190KAI | CLI Integration | Cloud Computing | Code review | Evaluation FrameworksCareer coaching and support | In-office culinary options | Inclusive family building benefits | Mental health wellness and fitness benefits | Retirement savings plansSenior-level Full TimeNew York City R1d ago
-
Artificial Intelligence | Automation | LLM integrations | Langchain | Language ModelsContractor | Fully remoteSenior-level ContractLos Angeles R1d ago
-
Software Engineer AI/ML USD 112K-150KA/B | A/B Testing | AWS | Anomaly Detection | Automated testingDental insurance | Employee assistance program | Health and wellness programs | Health insurance | Retirement benefitsMid-level Full TimeEvendale, United States R1d ago
-
Sr Data Engineer USD 91K-163KAI/ML | Azure DevOps | CI/CD | Code review | Data Compliance401k contribution | Career development opportunities | Comprehensive benefits | Incentive and recognition programsSenior-level Full TimeEden Prairie, Minnesota, UNITED STATES; REMOTE R1d ago
-
Senior AI/ML Engineer - Remote (Central Time Zone) USD 91K-163KAWS App Runner | AWS AppSync | AWS Bedrock | AWS Lambda | Agents SDK401k contribution | Benefits package | Equity stock purchase | Incentive and recognition programs | Remote workSenior-level Full TimeEden Prairie, Minnesota, UNITED STATES; REMOTE R1d ago