AI Test Engineer
Baltimore, MD, United States
A USD 98K-150K (estimate) Mid-level Full Time
Tasks
- Build automated testing frameworks for AI applications
- Define and validate AI guardrails and fallback logic
- Design AI testing strategies
- Detect hallucinations bias and model drift risks
- Develop test cases for edge cases and prompt behavior
- Document testing results and communicate findings
- Execute manual and automated AI output validation
- Identify workflow failures and performance bottlenecks
- Integrate automated tests into CI/CD pipelines
- Measure model performance against success criteria
- Perform data validation across ingestion and processing pipelines
- Support root cause analysis for AI production issues
- Test AI agent workflows and orchestration logic
- Validate AI agent integrations with APIs and data services
- Validate training validation and test dataset quality
Perks/Benefits
Skills/Tech-stack
API Testing | AWS | Agile | Bias detection | CI/CD | Case Development | Cause analysis | Data Quality | Data Validation | Databricks | Distributed Systems | Integration Testing | Language Models | Large Language Models | Machine Learning | Model Drift | Model Evaluation | Prompt engineering | Regression testing | Root Cause Analysis | Root cause | SQL | Scenario testing | Test Case | Test Planning | Test automation | Test case development
Education
Bachelor of Arts | Bachelor of Engineering | Bachelor of Science
Roles
AI | AI Test Engineer | Engineer | Machine Learning Test Engineer | Test Engineer
Related jobs
-
Featured Feat. Principal Knowledge & Data Architect USD 174K-284KAWS Neptune | Canonicalization | Chunking | Cypher | DBTBenefits including health and wellness programs | Health, wellness, and retirement plansSenior-level Full TimeHeadquarters - Chevy Chase, MD R18d ago
-
Senior Product Engineer, Fullstack USD 165K-250KAPI Design | Amplitude | CI/CD | Data Modeling | GoHealth insurance | Startup equitySenior-level Full TimeSan Francisco3h ago
-
Availability Calculation | Building codes | Capacity Planning | Cause analysis | CommissioningMid-level Full TimeMesa, Arizona, USA8h ago
-
Product Lifecycle Electrical Engineer, Data Center Engineering - Electrical Products and Services (DCE-EPS) USD 116K-160KBuilding codes | Cause analysis | Commissioning | Construction drawings | Data Center EngineeringSenior-level Full TimeHerndon, Virginia, USA8h ago
-
Robotics Software Engineer BRL 120K-145KAutonomous Navigation | C++ | Controls systems | Embedded Systems | Hardware-in-the-loopCareer growth | Comprehensive benefits | MentorshipMid-level Full TimeNorth Reading, MA8h ago
-
Senior Data Engineer - Modern Data & AI Platforms USD 156K-204KAWS | Cortex | DBT | Data Governance | Data IngestionRemote workSenior-level Full TimeKansas City, MO, United States10h ago
-
Data Engineer - SPACECOM USD 110K-142KAPI Development | Apache Spark | Data Validation | Data Visualization | DatabricksOnsite SCIF WorkMid-level Full TimeColorado Springs, CO11h ago
-
Staff Software Engineer - AI Platform USD 175K-287KApache Beam | Apache Flink | Apache Spark | C# | C++Health and wellness programs | Time awaySenior-level Full TimeMountain View, CA, United States11h ago
-
Senior Software Engineer - AI Platform USD 144K-236KApache Beam | C# | C++ | CUDA | CUDA optimizationHealth and wellness programs | Time offSenior-level Full TimeMountain View, CA, United States11h ago
-
Mid-level Full TimeWashington, D.C.12h ago
-
Data Engineer - CDAO Central USD 122K-142KAPI Development | Apache Spark | Cloud Computing | Data Pipelines | Data ValidationOnsite work | SCIF environment access | Top Secret SCI clearanceMid-level Full TimeWashington, D.C.13h ago
-
Principal AI Engineer - Aerospace & Defense USD 150K-180KAgentic architecture | Artificial Intelligence | Automated testing | CI/CD | CRM Integration401k company contribution | Community involvement | Disability benefits | Flexible paid time off | Flexible spending accountsSenior-level Full TimeItasca, Illinois, United States15h ago
-
AI Solutions Engineer USD 122K-168KAgentic AI | Bias Mitigation | CI/CD | Docker | Generative AICompany holidays | Health insurance | Learning and development reimbursement | Life insurance | Long-term disabilityMid-level Full TimeTorrance, California, United States15h ago
-
Data Engineer USD 100K-115KCrystal Reports | Data Warehouse | ETL | Oracle | Power BI401k matching | Accidental insurance | Critical illness insurance | Employee discount | Flexible spending accountMid-level Full TimeRomeoville, IL, United States15h ago
-
AI Engineer III USD 120K-140KAgent systems | Agentic AI | Amazon S3 | Amazon Web Services | Apache Airflow401k matching | Dental insurance | Employee assistance program | Flexible PTO | Life insuranceSenior-level Full TimeSan Antonio, TX, United States15h ago
-
AI/ML Systems Engineer USD 135K-210KAI Assurance | AI Governance | AI machine learning | Agentic Workflows | Anomaly DetectionSenior-level Full TimeDayton, OH, USA16h ago
-
Data Engineer - TRANSCOM USD 100K-135KAPI Development | Apache Spark | Data Pipeline Development | Data Validation | Data pipelineOnsite SCIF Work | Top Secret clearance required to startMid-level Full TimeScott AFB, IL16h ago
-
AI Agents | AI-assisted coding | AWS | Artificial neural networks | Assisted codingFully remote | Occasional onsite visitsSenior-level Full TimeColumbia, SC, United States R16h ago
-
Data Engineer USD 114K-160KCribl | Data Modeling | Data Warehouse | Data analytics | Data pipeline401k match | Holiday pay | Medical, dental & vision coverage | Paid time offMid-level Full TimeHuntsville, AL, United States16h ago
-
Machine Learning Engineer, GenAI Technology USD 180K-300KArtificial Intelligence | C++ | Compliance | Data integration | Java401k match | Employee affinity groups | Fully-paid health care benefits | Generous parental and family leave | Mental wellness programsSenior-level Full TimeNew York, NY17h ago
-
20454 - Analytics Engineer x 2 GBP 40K-42KAzure Data | Azure Data Factory | Azure Synapse | Data Factory | Data ModelingFlexible working | Hybrid working | Training and coachingEntry-level Full TimeNational, National18h ago
-
Senior Data Engineer USD 92K-135KAWS | Azure Data | Azure Data Factory | Change Data Capture | Cloud Security401k matching | Lifecycle benefits | Loan forgiveness | Tuition reimbursementSenior-level Full TimeChicago, IL, United States18h ago
-
Airflow | Apache Spark | CI/CD | Data Governance | Data LakehouseSenior-level ContractAustin, United States18h ago
-
Causal Inference | Cross-modal fusion | DPO | Data Modeling | Fraud DetectionEntry-level Full TimeSan Jose, California, United States19h ago
-
CV/NLP/Multimodal LLM Machine Learning Engineer Graduate (TikTok Trust and Safety) - 2026 Start (BS/MS) USD 119K-177KActive Learning | Computer Vision | Data Pipelines | Language Models | Language ProcessingGrowth opportunities | Onboarding programEntry-level Full TimeSan Jose, California, United States19h ago