AI-Factory Observability Principal
Tasks
- Correlate network behavior with training performance
- Define SLOs alerting strategy and on call signal quality
- Define observability architecture
- Design unified data model for correlation
- Establish instrumentation and telemetry pipelines
- Generate incident summaries with LLMs
- Implement AIOps anomaly detection and forecasting
- Instrument Kubernetes for AI workloads
- Integrate BMS telemetry into observability platform
- Integrate EPMS power telemetry into observability platform
- Mentor engineers and set observability roadmap direction
- Monitor GPU accelerator clusters
- Own metrics logs traces and events
Perks/Benefits
- N/A
Skills/Tech-stack
AIOps | Anomaly Detection | BACnet | BMS | DNP3 | Datacenter Engineering | Datadog | Dynatrace | ECC errors | EPMS | Elasticsearch | Events | Forecasting | GNMI | GPU Monitoring | Grafana | IEC 61850 | IPMI | Infiniband | Infrastructure as Code | Kubernetes | Kubernetes Scheduler | LLM | Language Processing | Linux | Logs | Loki | Machine Learning | Metrics | Modbus | Mqtt | NVIDIA DCGM | NVLink | Natural Language | Natural Language Processing | Netflow | OPC UA | Observability | OpenTelemetry | Openconfig | Predictive Maintenance | Prometheus | Python | RDMA | Redfish | RoCE | SFlow | SNMP | SRE | Splunk | Streaming telemetry | Telemetry pipelines | Tempo | Terraform | Thanos | Time Series | Time-series databases | Traces | VictoriaMetrics | WMI | “as-code”
Education
N/A
Related jobs
-
Featured Feat. Principal Knowledge & Data Architect USD 174K-284KAWS Neptune | Canonicalization | Chunking | Cypher | DBTBenefits including health and wellness programs | Health, wellness, and retirement plansSenior-level Full TimeHeadquarters - Chevy Chase, MD R14d ago
-
Featured Feat. Associate Director, Data Labs USD 167K-167KAWS | Cloud Computing | Compute Infrastructure | Data Analysis | LLM GovernanceConference speaking opportunities | Hybrid work schedule | Media appearancesSenior-level Full TimeWashington, District of Columbia, 20004, United … R27d ago
-
Mid-Level Data Engineer USD 90K-98KAPI Development | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake StorageContract award contingent | Remote workMid-level Full TimeWork from home, VA, United States R5h ago
-
Senior-level Full TimeWork from home, VA, United States R5h ago
-
Sr. Databricks Consultant USD 175K-250KAWS S3 | Apache Spark | Azure Data | Azure Data Lake | CI/CDRemote work | Secret clearance | Travel as neededSenior-level Full TimeWork from home, VA, United States R5h ago
-
AI Engineer USD 150K-210KEvaluation systems | LLM APIs | Language Models | Large Language Models | PostgreSQL401k plan | Flexible time off | Flexible work schedules | Health and wellness benefits | In-person offsitesMid-level Full TimeSan Francisco, CA / Remote (US) R7h ago
-
Senior DevOps Engineer (Storage) - remote, US/EU USD 202K-290KAir-gapped | Air-gapped environments | Argo CD | ArgoCD | BashConference attendance | Hackathons | Professional development and training | Remote work | Tech talksExecutive-level Full TimeAustin, TX, United States R14h ago
-
AI Engineer USD 113K-180KAngular | Apache Kafka | CI/CD | Context Management | DevOps401k matching | Charitable matching | Community outreach groups | Flexible paid time off | Life insuranceMid-level Full TimeLeawood, KS, US R16h ago
-
Machine Learning Engineer USD 131K-177KAWS | Cassandra | Classification | Cloud Migration | Computer VisionLearning opportunities | Professional development | Work from homeMid-level Full TimeRemote, NY, US R17h ago
-
Senior Data Engineer (DBA) USD 91K-163KAmazon Web Services | Ansible | Azure | Capacity Planning | Cloud platformCareer development | Comprehensive benefits package | On-call rotation | Telecommute flexibilitySenior-level Full TimeHartford, Connecticut, UNITED STATES; REMOTE R21h ago
-
Sr. Data Engineer - Remote USD 91K-163KAzure Data | Azure Data Factory | Batch Processing | CI/CD | Data Factory401k contribution | Comprehensive benefits package | Equity stock purchase | Flexible work schedule | Incentive and recognition programsSenior-level Full TimeLittle Rock, Arkansas, UNITED STATES; REMOTE R21h ago
-
Senior AI ML Software Engineer - Data Platforms - Remote Nationwide or Hybrid in MN or DC USD 120K-214KAI Agents | AWS | Artificial Intelligence | Automated testing | AzureSenior-level Full TimeEden Prairie, Minnesota, UNITED STATES; REMOTE R21h ago
-
Senior Principal Data Scientist - Remote USD 134K-230KAgentic Systems | Convolutional Neural Networks | Deep learning | Gradient boosting | Language ModelsRemote work flexibilitySenior-level Full TimeSan Diego, California, UNITED STATES; REMOTE R21h ago
-
NLP Engineer USD 72K-130KArtifactory | Artificial Intelligence | C# | CI/CD | Code review401k contribution | Comprehensive benefits | Equity stock purchase | Incentive and recognition programs | Telecommute OptionsMid-level Full TimeSan Diego, California, UNITED STATES; REMOTE R21h ago
-
Senior Data Science Engineer USD 116K-198KApache Airflow | CI/CD | DBT | Databricks | Databricks AutoloaderFlexible work optionsSenior-level Full TimeRemote (United States) R22h ago
-
Senior Embedded Software Engineer - Future Forward USD 134K-201KAuthentication | C# | C++ | CAN | CUDASenior-level Full TimeSunnyvale, CA, United States R1d ago
-
Research Operations, Code USD 130K-250KData pipeline | Experiment design | LLM | Language Processing | Machine LearningDental insurance | Equity grant | Free fitness membership | Health insurance | Housing bonusSenior-level Full TimeSan Francisco R1d ago
-
Senior SoC Power Analysis & Optimization Engineer USD 120K-600KAI workload | AI workload characterization | ASIC | Clock gating | Gate Level Analysis401k match | Company holidays | Flexible spending accounts | Health, dental, vision, and life insurance | Home tech setup reimbursementSenior-level Full TimeMountain View, CA R1d ago
-
Azure Data Engineer USD 110K-140K3NF | API Management | Agile Scrum | Azure Data | Azure Data Factory401k match | Employee resource group network | Holiday pay | Long-term disability | Medical, dental & vision coverageSenior-level Full TimeUnited States R1d ago
-
Production Support Senior Analyst USD 113K-151KAPI Integration | APIs | Agile | Backlog Management | Dashboard DevelopmentHealth and life insurance | Hybrid work model | Paid time off | Personal and family care leaves | Retirement benefitsSenior-level Full TimeJersey City, NJ, United States R1d ago
-
Lead AI Engineer USD 130K-201KAI Search | App Service | Azure AI | Azure AI Search | Azure App401k match | Employee Resource Group | Health insurance | Paid time off | Parental leaveSenior-level Full TimeUnited States R1d ago
-
Senior AI Engineer (m/f/d) USD 100K-140K.NET | AI Search | AIOps | API Integration | AWS100 percent salary during sick leave | 4 weeks remote work per year | Childcare support | Coursera access | Experience daysSenior-level Full TimeMadrid, MD, Spain R1d ago
-
Sr. Forward Deployed Engineer-Retail USD 182K-250KAI APIs | AWS | Apache Spark | Artificial Intelligence | AzureCustomer site travelSenior-level Full TimeRemote - California R1d ago
-
Data Engineer USD 105K-140KAPIs | AWS | Apache Airflow | Apache Iceberg | Apache Kafka401k plan with company match | Annual bonus | Company equipment provided | Company events and team outings | Company medical dental and vision plansMid-level Full TimeAtlanta, GA preferred, Remote R1d ago
-
Behavioral Data | Data Pipelines | Data Quality | Distributed Systems | Evaluation metricsGlobal annual leave policy | In-person onboarding | Remote work | Restricted stock units (RSUs)Senior-level Full TimeUnited States (Remote) R1d ago