Manager, AI Engineering (Tester )
O'Fallon, Missouri (Main Campus), United States
USD 140K-231K Mid-level Full Time
Tasks
- Assess chunk relevance
- Benchmark semantic performance
- Build agentic AI test suites
- Build automated prompt regression pipelines
- Build production monitoring pipelines
- Check answer grounding
- Define AI testing roadmap and standards
- Design LLM evaluation frameworks
- Design inference performance benchmarks
- Detect embedding shifts
- Detect hallucinations
- Detect retrieval degradation
- Detect semantic output drift
- Document evaluation metrics and tooling
- Enforce LLM quality gates in CI/CD
- Evaluate RAG pipelines
- Execute red teaming
- Handle agent failures
- Measure latency and throughput
- Measure retrieval precision
- Measure token efficiency
- Monitor anomalous agent behaviors
- Perform adversarial testing
- Run fairness bias audits
- Run responsible AI audits
- Score LLM outputs
- Test context poisoning
- Test data leakage
- Test jailbreaks
- Test model manipulation
- Test prompt injection
- Track quality degradation
- Validate context faithfulness
- Validate goal completion
- Validate inter agent coordination
- Validate task decomposition
- Validate tool selection
Perks/Benefits
- N/A
Skills/Tech-stack
AWS | Adversarial Testing | Anthropic | Azure | CI/CD | Cloud platform | CloudWatch | Databricks | Datadog | DeepEval | Google Cloud | Google Cloud Platform | Grafana | Hallucination detection | Hugging Face | LLMOps | Langchain | Langgraph | Langsmith | MLOps | MLflow | OpenAI | Prompt injection | PromptFlow | Python | Ragas | Red Teaming | Retrieval-Augmented Generation | SQL | SageMaker | TruLens | Weights and Biases
Education
Bachelor of Engineering | Bachelor of Science | Master of Science
Roles
AI | AI Engineering | AI Engineering Manager | AI Test Engineer | Engineer | Engineering Manager | Manager | Test Engineer
Related jobs
-
Associate Consultant, Generative AI USD 84K-122KAWS | Azure | CI/CD | Cloud platform | Compliance401k match | Employee assistance program | Flexible spending account | Health savings account | Paid parental leaveMid-level Full TimeSan Francisco, CA, United States6h ago
-
Sr AI Engineer USD 84K-105KAudio signal processing | C Embedded Systems | C embedded | C# | Data Preprocessing100% onsite | ADND Insurance | Accrued Paid Vacation | Company paid life insurance | Dental insuranceSenior-level Full TimeColumbia, MARYLAND, United States8h ago
-
Data Engineer USD 100K-160KAirflow | Amazon Redshift | Apache Spark | Azure | Azure Data401k retirement plan | Employee assistance program | Health savings account | Life insurance | Long-term disabilityMid-level ContractCoppell, Texas, USA8h ago
-
AI Engineer (Remote) USD 169K-230KAWS | Agent systems | Application development | Automation | Azure401k plan | Accident insurance | Dental insurance | Disability insurance | Flexible spending accountExecutive-level Full TimeFort Washington, PA, United States R11h ago
-
Mid-Level Data Engineer USD 130K-140KAPI first | API-first design | AWS | Agile | Amazon RedshiftRemote workMid-level Full TimeWork from home, VA, United States11h ago
-
Sr. Data Engineer USD 150K-160KAWS | Agile | Automated testing | Big Data | Continuous integrationRemote workSenior-level Full TimeWork from home, VA, United States11h ago
-
Data Visualization Engineer USD 102K-127KAI | Access Control | Data Governance | Data Modeling | Data StudioBereavement | Dental insurance | Disability insurance | Employee assistance program | Employee discount programSenior-level Full TimeRemote - Nationwide, United States11h ago
-
Data Engineer IV USD 110K-155KAWS | Azure | Azure Data | Azure Data Factory | CI/CD401k | Accident insurance | Continuing education assistance | Critical illness insurance | Employee assistance programSenior-level Full TimeNorthern Virginia, VA; Northern Virginia, VA; …12h ago
-
Data Engineer III USD 95K-130KAWS | Azure | Azure Data | Azure Data Factory | CI/CD401k | Accident insurance | Continuing education assistance | Critical illness insurance | Employee assistance programSenior-level Full TimeNorthern Virginia, VA; Northern Virginia, VA12h ago
-
Data Engineer II USD 75K-105KData Pipelines | Data Quality | Data Validation | Data Warehousing | Data integration401k | Accident insurance | Continuing education support | Critical illness insurance | Employee assistance programEntry-level Full TimeNorthern Virginia, VA; Northern Virginia, VA; …12h ago
-
Senior Technical Product Manager – Data Platforms & Internal Products (Remote – U.S.) - 1807 A USD 180K-230KAPI | AWS | Amazon Redshift | Apache Spark | AutomationEquity | Remote workEntry-level Full TimeSan Francisco, CA R12h ago
-
Azure Data | Azure Data Factory | Azure DevOps | Azure Machine Learning | CI/CDRemote | USC OnlySenior-level Full TimeDallas, TX. R12h ago
-
Data Scientist / Machine Learning Engineer USD 110K-150KAPIs | Agile | CI/CD | Computer Vision | Continuous Deployment401k | Dental insurance | Flexible spending account | Health insurance | Life and AD&D insuranceMid-level Full TimeSan Diego, California; San Diego, CA; …12h ago
-
ABAC | AWS CloudFormation | AWS Lambda | AWS Step Functions | AWS-APIsOnsite/hybrid workSenior-level Full TimeChantilly, Virginia R12h ago
-
Sr. AI Developer, Engineering USD 150K-189K.NET | AI Evaluations | APIs | Adobe Experience | Adobe Experience CloudSenior-level Full TimeChicago, IL, United States13h ago
-
AI Agents | AI Workflow Orchestration | AI workflow | AWS | AWS LambdaDental | Medical | Paid time off (PTO) | Remote work | RetirementSenior-level Full TimeLos Angeles, CA, United States13h ago
-
AWS Lambda | AWS Step Functions | Amazon Web Services | Anthropic | Cloud IntegrationsDental insurance | Medical insurance | Paid time off | Remote work | Retirement benefitsMid-level Full TimeIrvine, CA, United States13h ago
-
GenAI Chemical Safety Expert USD 150K-178KAI Safety | Adversarial Testing | Analytical Chemistry | Chemical Security | Chemical Weapons ConventionSenior-level Full TimeNew York, NY, US R14h ago
-
GenAI Biosecurity Expert USD 150K-178KBiological Risk | Biological Risk Assessment | Biosafety | Biosecurity | CRISPRSenior-level Full TimeNew York, NY, US R14h ago
-
GenAI Biosecurity Expert USD 150K-178KAdversarial prompting | Biological Risk | Biological Risk Assessment | Biomanufacturing | BioreactorSenior-level Full TimeNew York, NY, US R14h ago
-
Sr Data Analytics Mgr. Paid Media Campaigns USD 139K-244KAccount-based marketing | Adtech | Attribution Modeling | Data Quality | KPI governance401k plan | ESPP | Family leave programs | Flexible spending accounts | Flexible time awaySenior-level Full TimeSan Diego, California, United States R14h ago
-
AUTOSAR | C++ | CAN | DaVinci Developer | Hardware-in-the-loopSenior-level Full TimeDearborn, United States14h ago
-
AI Developer Builder USD 140K-180KAWS | Agentic Systems | Anthropic Bedrock | Context pipelines | DatadogDirect input on roadmap | High ownershipSenior-level Full TimeTel Aviv-Yafo, Tel Aviv District, IL R15h ago
-
Artificial Intelligence Machine Learning Engineer USD 98K-163KArtificial Intelligence | CI/CD | Docker | Elasticsearch | FastAPIHealth insurance | Holiday pay | Learning and development | Life insurance | Long-term disabilitySenior-level Full TimeUSA-VA-Ashburn15h ago
-
.NET | A/B | A/B Testing | AI Agent | AI agent orchestrationMid-level Full TimeDallas, Texas, United States; Hermitage, Tennessee, …16h ago