Principal Tools, Automation & Testing Engineer
Tasks
- Build alerting systems for model issues
- Build automated pipelines for model validation regression testing and benchmarking
- Create evaluation datasets synthetic data and edge case scenarios
- Debug model behavior and identify root causes of failures
- Define test strategies with engineers and product teams
- Design and develop testing frameworks for AI ML models
- Develop tools for prompt testing output validation and hallucination detection
- Ensure compliance with ethical AI standards fairness and bias testing
- Implement metrics for accuracy robustness latency and safety
- Monitor model performance in production
Perks/Benefits
- N/A
Skills/Tech-stack
AI Safety | APIs | AWS | Adversarial Testing | Algorithms | Azure | Bias detection | CI/CD | Data Generation | Data Structures | Distributed Systems | GCP | LLMs | Langchain | Machine Learning | Model Evaluation | Monitoring | Observability | OpenAI evals | Prompt engineering | Pytest | Python | Software Design | Synthetic Data Generation | Synthetic data | Unittest
Education
Bachelor of Engineering | Bachelor of Science | Master of Science
Regions
Countries
States
Cities
Related jobs
-
Featured Feat. Applied AI Engineer - Bay Area USD 211K-263KArtificial Intelligence | C plus plus | C# | Embeddings | Feature Engineering401k | Comprehensive health and wellness benefits | Learning and development opportunities | Unlimited time offMid-level Full TimeHQ (San Francisco)24d ago
-
Senior Data Engineer (Remote) USD 155KAgile | Apache Spark | BigQuery | Cassandra | Data Governance401k match | Dental insurance | Employee assistance program | Employee stock purchase plan | Flexible scheduleSenior-level Full TimeWork From Home, United States R6h ago
-
Senior AI Operations Engineer USD 170K-180KAI infrastructure | Azure | CI/CD | Cloud infrastructure | Container Engine for Kubernetes401k match | Employee assistance program | Employee stock purchase plan | Flexible schedule | Flexible spending accountSenior-level Full TimeWork From Home, United States R6h ago
-
API Development | Airflow | Automated retraining | CI/CD | Cloud PlatformsEquityMid-level Full TimeNaples, United States8h ago
-
Adversarial Machine Learning | Anomaly Detection | Cloud Security | Machine Learning | PythonSecurity clearance premiumsMid-level Full TimeNaples, United States8h ago
-
API Design | AWS | AWS Cloud | AWS Cloud Development Kit | AWS cloud developmentSenior-level ContractGlendale, United States9h ago
-
Mid-level Full TimeUS-Kansas-Wichita10h ago
-
Senior-level Full TimeCincinnati, OH, United States10h ago
-
Senior Data Scientist - Government & Public Services USD 113K-208KClass imbalance | Cloud Computing | Data Exploration | Data Preparation | Data leakageSenior-level Full TimeArlington/Rosslyn, Virginia, United States10h ago
-
Delivery Senior Consultant, Data Engineering and Gen AI USD 119K-208K.NET | AWS | Agentic AI | Agile | AngularSenior-level Full TimeGilbert, Arizona, United States; Lake Mary, …10h ago
-
Software Engineer/Researcher, AI-Native Database Systems USD 156K-387KC++ | Database Architecture | Distributed Systems | Indexing | Information RetrievalSenior-level Full TimeSan Jose, California, United States10h ago
-
Software Engineer Level 1 -FFNN-8889 USD 78K-250KAccumulo | BSON | Bigtable | Distributed Systems | HBase401k match | Employee referral programs | FSA | Flexible work arrangements | Mental health supportMid-level Full TimeHanover, MD10h ago
-
Software Engineer Level 2 -FFNN-8890 USD 78K-250KAccumulo | BSON | Bigtable | Database Design | Development Lifecycle401k match | Dental insurance | Employee referral programs | Flexible spending accounts | Flexible work arrangementsMid-level Full TimeHanover, MD10h ago
-
Data Pipelines | Data Storage | Distributed Systems | High Performance | High-Performance ComputingCareer growthEntry-level Full TimeSan Jose, California, United States10h ago
-
Agent architecture | Backend Development | Document ingestion | Frontend Development | IndexingCross-functional collaboration | Hands-on experience | MentorshipEntry-level InternshipSan Jose, California, United States10h ago
-
Apache Flink | Apache Spark | Automation | C++ | Cause analysisSenior-level Full TimeSan Jose, California, United States10h ago
-
Cost estimation | Distributed Caches | Distributed Systems | Document Databases | Embedding IngestionSenior-level Full TimeSeattle, Washington, United States10h ago
-
Research Engineer / Scientist - Storage for LLM USD 156K-387KAttention Mechanisms | CUDA | Caching | Distributed Systems | Eviction policiesCompetitive compensation | Conference attendance | Generous research resources | Innovation-driven culture | Open source contributionsEntry-level Full TimeSan Jose, California, United States11h ago
-
Machine Learning Engineer Intern (E-Commerce Recommendation/Search Alliance) - 2026 Start (PhD) USD 129K-246KComputer Vision | Data Analysis | E-commerce | E-commerce Data | Language ProcessingEntry-level InternshipSeattle, Washington, United States11h ago
-
Agentic data | Apache Hive | Apache Spark | Coding Data | Data CurationSenior-level Full TimeMenlo Park, CA11h ago
-
Prototype Process Engineer- Robotics Studio USD 139K-200KAgent Orchestration | Assembly line | Assembly line operations | Bias Mitigation | Electromechanical SystemsMid-level Full TimeRedmond, WA | Burlingame, CA11h ago
-
Mechanical Engineer, Data Center Design Engineering USD 171K-248KAI tool integration | Acoustics Engineering | Agent Orchestration | Airside Systems | Availability EngineeringSenior-level Full TimeMenlo Park, CA | San Francisco, …11h ago
-
Staff Software Engineer, Agentic AI, Trust and Safety USD 207K-301KAgentic AI | Anti-abuse | Anti-abuse systems | Architecture ownership | Artificial IntelligenceSenior-level Full TimeKirkland, WA, USA11h ago
-
Cloud Data and AI Engineer, Professional Services USD 127K-183KBigtable | C++ | Cloud Databases | Cloud SQL | Cloud platformTravel up to 30%Mid-level Full TimeReston, VA, USA11h ago
-
Senior Software Engineer, AI/ML, Google Cloud AI USD 174K-253KC++ | Data Processing | Data Structures | Data Structures and Algorithms | DebuggingSenior-level Full TimeKirkland, WA, USA; Sunnyvale, CA, USA11h ago