Principal AI Tool Engineer for Software Automation/Testing
Tasks
- Build automated pipelines for model validation regression testing and benchmarking
- Collaborate to define test strategies
- Create evaluation datasets and synthetic data for edge cases
- Debug model behavior and identify root causes of failures
- Design and develop testing frameworks for AI ML models and LLM applications
- Develop prompt testing output validation and hallucination detection tools
- Ensure compliance with ethical AI standards fairness and bias testing
- Implement metrics for accuracy robustness latency and safety
- Monitor model performance in production and build alerting systems
Perks/Benefits
- N/A
Skills/Tech-stack
API | AWS | CI/CD | Cloud platform | Distributed Systems | Google Cloud | Google Cloud Platform | Langchain | Language Models | Language Processing | Large Language Models | Machine Learning | Microsoft Azure | Model Evaluation | Monitoring | Natural Language | Natural Language Processing | Observability | OpenAI evals | Prompt engineering | Pytest | Python | Synthetic data | Unittest
Education
Roles
AI | AI Test Engineer | Engineer | Principal | Principal AI Tool Engineer | Test Engineer | Tool Engineer
Regions
Countries
States
Related jobs
-
C++ | CUDA | Data Preprocessing | Data Processing | Data loadingSenior-level Full TimeUS, CA, Santa Clara1d ago
-
C# | C++ | Caching | Distributed Systems | GPU ArchitectureComprehensive benefits | EquitySenior-level Full TimeUS, CA, Santa Clara R1d ago
-
Senior Deep Learning Frameworks Sustaining Engineer USD 152K-218KBazel | C# | C++ | CI Jenkins | CI/CDBenefits | EquitySenior-level Full TimeUS, CA, Santa Clara1d ago
-
Senior AI Workflow Engineer USD 184K-287KASP.NET | Agentic AI | Ansible | Artifactory | CI/CDEquity | Health insurance | Paid time offSenior-level Full TimeUS, CA, Santa Clara1d ago
-
AWS | Agent systems | Azure | C plus plus | Capacity PlanningSenior-level Full TimeUS, CA, Santa Clara1d ago
-
Senior Data Engineer, Engineering Data Analytics USD 168K-270KAWS Glue | Amazon Athena | Amazon EMR | Amazon Redshift | Amazon S3Equity | Health benefitsSenior-level Full TimeUS, CA, Santa Clara1d ago
-
Senior Software Engineer, CUDA Deep Learning Systems USD 184K-356KC++ | CUDA | Computer Architecture | Deep Learning Compilers | Deep learningBenefits | EquitySenior-level Full TimeUS, CA, Santa Clara R1d ago
-
Benchmarking | C++ | C++14 | CUDA | Compiler developmentSenior-level Full TimeUS, CA, Santa Clara1d ago
-
Algorithms | C++ | CUDA | Compilers | Computer ArchitectureSenior-level Full TimeUS, CA, Santa Clara1d ago
-
Azure DevOps | Bazel | C plus plus | C# | CI/CDSenior-level Full TimeUS, CA, Santa Clara1d ago
-
Software Engineer, CUDA Deep Learning Systems USD 124K-195KC++ | CUDA | Code generation | Computer Architecture | Deep Learning CompilersEquity | Health benefitsMid-level Full TimeUS, CA, Santa Clara R1d ago
-
Senior Deep Learning Algorithm Engineer USD 152K-241KAutoscaling | C++ | Deep learning | Distributed Systems | Inference OptimizationSenior-level Full TimeUS, CA, Santa Clara R1d ago
-
Senior Software Engineer, Machine Learning Inference USD 152K-241KC++ | CUDA | Compilers | Deep learning | GPU ProgrammingSenior-level Full TimeUS, CA, Santa Clara1d ago
-
Senior Machine Learning Engineer USD 184K-356KC++ | Computer Vision | Data pipeline | Deep learning | LidarSenior-level Full TimeUS, CA, Santa Clara1d ago
-
Senior Perception Engineer - Autonomous Vehicles USD 184K-356KC++ | Computer Vision | Data Pipelines | Deep learning | LidarBenefits | EquitySenior-level Full TimeUS, CA, Santa Clara1d ago
-
Senior Storage Production Engineer - DGX Cloud USD 184K-333KAccess Control | Anomaly Detection | Ansible | Auditing | BashSenior-level Full TimeUS, CA, Santa Clara R1d ago
-
Engineering Manager, AI Developer Technology USD 184K-356KC plus plus | CPU architecture | CUDA | Computer Architecture | Deep learningEmployee benefits | Equity | Hybrid workSenior-level Full TimeUS, CA, Santa Clara1d ago
-
Senior AI Systems and Algorithms Engineer USD 152K-241KDeep learning | Distributed Systems | Distributed Training | FSDP | GPU PerformanceSenior-level Full TimeUS, CA, Santa Clara R1d ago
-
C++ | CPU architecture | CUDA | Deep learning | Fraud DetectionSenior-level Full TimeUS, CA, Santa Clara R1d ago
-
C++ | Contact mechanics | Control Systems | Deep learning | DrakeMid-level Full TimeSanta Clara2d ago
-
C++ | Computer Science | Computer Vision | Contact mechanics | ControlsMid-level Full TimeSanta Clara2d ago
-
Automation | C++ | CUDA | Computer Architecture | ContainersSenior-level Full TimeUS, CA, Santa Clara R2d ago
-
Senior Storage Software Engineer - DGX Cloud USD 224K-356KBenchmarks | Block layer | C# | C++ | CSI driversBenefits | EquitySenior-level Full TimeUS, CA, Santa Clara R2d ago
-
Bash | Bootstrap | CSI | CSS3 | Container StorageSenior-level Full TimeUS, CA, Santa Clara R2d ago
-
AI and ML Power Methodology Engineer USD 136K-218KAgentic Frameworks | Algorithms | C++ | Data Annotation | Data PipelinesBenefits | EquitySenior-level Full TimeUS, CA, Santa Clara2d ago