Principal AI Tool Engineer for Software Automation/Testing
Tasks
- Build automated pipelines for model validation regression testing and benchmarking
- Collaborate to define test strategies
- Create evaluation datasets and synthetic data for edge cases
- Debug model behavior and identify root causes of failures
- Design and develop testing frameworks for AI ML models and LLM applications
- Develop prompt testing output validation and hallucination detection tools
- Ensure compliance with ethical AI standards fairness and bias testing
- Implement metrics for accuracy robustness latency and safety
- Monitor model performance in production and build alerting systems
Perks/Benefits
- N/A
Skills/Tech-stack
API | AWS | CI/CD | Cloud platform | Distributed Systems | Google Cloud | Google Cloud Platform | Langchain | Language Models | Language Processing | Large Language Models | Machine Learning | Microsoft Azure | Model Evaluation | Monitoring | Natural Language | Natural Language Processing | Observability | OpenAI evals | Prompt engineering | Pytest | Python | Synthetic data | Unittest
Education
Roles
AI | AI Test Engineer | Engineer | Principal | Principal AI Tool Engineer | Test Engineer | Tool Engineer
Regions
Countries
States
Related jobs
-
Applied AI Innovation Engineer — Sales AI Applications USD 124K-195KAI Governance | APIs | Agentic Workflows | Artificial Intelligence | BenchmarkingEmployee benefits | EquityMid-level Full TimeUS, CA, Santa Clara1d ago
-
Agile | Automation tooling | CUDA | Capacity Planning | Compute Capacity Planning401k match | Dental insurance | Employee stock purchase plan | Equity | Flexible paid time offSenior-level Full TimeUS, CA, Santa Clara1d ago
-
Checkpointing | Data Pipelines | Data Processing | Data loading | Data parallelismSenior-level Full TimeUS, CA, Santa Clara R1d ago
-
Machine Learning Engineer, AI Safety USD 152K-270KAlignment | Automation | Bias detection | Content Safety | Dataset developmentMid-level Full TimeUS, CA, Santa Clara R1d ago
-
AI Quality and Agent Operations Analyst - AI Solutions USD 124K-195KAPIs | Case design | Cause analysis | Data Analysis | Data workflowsMid-level Full TimeUS, CA, Santa Clara1d ago
-
Lead AI Data Engineer – Platform & Analytics BRL 40K-40KAWS Athena | Apache Airflow | Apache Iceberg | DAGs | DBT CoreSenior-level Full TimeUS,CA,Santa Clara2d ago
-
Senior Software Engineer - AI Inference Performance USD 184K-287KBatching | C++ | CUDA | CUDA Graphs | CUDA kernelsSenior-level Full TimeUS, CA, Santa Clara2d ago
-
C# | C++ | Cloud Computing | Data Management | Distributed SystemsSenior-level Full TimeUS, CA, Santa Clara2d ago
-
AWS | Azure | C++ | Capacity Planning | Data ModelingSenior-level Full TimeUS, CA, Santa Clara2d ago
-
Distributed Training | Experiment design | GPU Acceleration | Language Models | Large Language ModelsMid-level Full TimeUS,CA,Santa Clara2d ago
-
Senior-level Full TimeUS, CA, Santa Clara R3d ago
-
Automation | CI/CD | Command Line | Command-line Interface | Continuous integrationEquity | Hybrid workSenior-level Full TimeUS, CA, Santa Clara3d ago
-
Senior Applied AI Engineer, Product Simulation USD 184K-287KAI | Agent Design | CI/CD | Firmware | Human-in-the-loopSenior-level Full TimeUS, CA, Santa Clara3d ago
-
Lead ASIC Package Design Automation and AI Engineer USD 160K-264K2.5D IC | 3D-IC | Artificial Intelligence | Auto-routing | C++Senior-level Full TimeUS,CA,Santa Clara3d ago
-
Staff/Sr. Staff AI Engineer - IT Business Applications USD 145K-235KAWS | Agile | Azure | CI/CD | Distributed SystemsSenior-level Full TimeSanta Clara, CA4d ago
-
AWS | Adversarial Training | Agent systems | Autonomous Agents | AzureSenior-level Full TimeSanta Clara, CA4d ago
-
Senior Deep Learning Software Engineer, DLSim USD 152K-241KArchitectural simulation | C# | C++ | Compiler infrastructure | Compiler optimizationEquity | Health benefits | Paid time offSenior-level Full TimeUS, CA, Santa Clara4d ago
-
Architecture Search | CUDA | CUDA Compilation | CUDA kernel | GPU ProfilingSenior-level Full TimeUS, CA, Santa Clara R4d ago
-
Senior Software Engineer - Cloud Data Platform USD 140K-160KAWS | Alerting | Azure | CI/CD | Cause analysis401k | Dental insurance | Employee referral program | Health savings account | Life insuranceSenior-level Full TimeSanta Clara, CA6d ago
-
Software Engineer (Applied AI/ML) USD 200K-250KCI/CD | Data Pipelines | Data ontologies | Design of Experiments | Digital Twins401k | Bonuses | Equity | FSA | Flexible time offMid-level Full TimeSanta Clara, CA or Remote R6d ago
-
Principal Software Engineer (Data Platform Prisma AIRS) USD 147K-237KAirflow | Apache Beam | Apache Spark | Batch Processing | BenchmarkingSenior-level Full TimeSanta Clara, CA7d ago
-
Senior Solutions Architect, Inference Service Providers USD 184K-287KAgent Design | Cache Management | Custom Inference Kernels | Data Curation | DevOpsSenior-level Full TimeUS, CA, Santa Clara R7d ago
-
Array computing | Asynchronous programming | C++ | CUDA | Compiler optimizationBenefits | EquitySenior-level Full TimeUS, CA, Santa Clara R7d ago
-
Senior Software Engineer, RL Post-Training Frameworks USD 184K-287KActor Based Programming | C# | C++ | Deep learning | Distributed SystemsSenior-level Full TimeUS, CA, Santa Clara R7d ago
-
AWS | Azure | C# | C++ | Computer VisionEntry-level InternshipUS,CA,Santa Clara R7d ago