Manager, AI Engineering (Tester )
O'Fallon, Missouri (Main Campus), United States
USD 140K-231K Mid-level Full Time
Tasks
- Assess chunk relevance
- Benchmark semantic performance
- Build agentic AI test suites
- Build automated prompt regression pipelines
- Build production monitoring pipelines
- Check answer grounding
- Define AI testing roadmap and standards
- Design LLM evaluation frameworks
- Design inference performance benchmarks
- Detect embedding shifts
- Detect hallucinations
- Detect retrieval degradation
- Detect semantic output drift
- Document evaluation metrics and tooling
- Enforce LLM quality gates in CI/CD
- Evaluate RAG pipelines
- Execute red teaming
- Handle agent failures
- Measure latency and throughput
- Measure retrieval precision
- Measure token efficiency
- Monitor anomalous agent behaviors
- Perform adversarial testing
- Run fairness bias audits
- Run responsible AI audits
- Score LLM outputs
- Test context poisoning
- Test data leakage
- Test jailbreaks
- Test model manipulation
- Test prompt injection
- Track quality degradation
- Validate context faithfulness
- Validate goal completion
- Validate inter agent coordination
- Validate task decomposition
- Validate tool selection
Perks/Benefits
- N/A
Skills/Tech-stack
AWS | Adversarial Testing | Anthropic | Azure | CI/CD | Cloud platform | CloudWatch | Databricks | Datadog | DeepEval | Google Cloud | Google Cloud Platform | Grafana | Hallucination detection | Hugging Face | LLMOps | Langchain | Langgraph | Langsmith | MLOps | MLflow | OpenAI | Prompt injection | PromptFlow | Python | Ragas | Red Teaming | Retrieval-Augmented Generation | SQL | SageMaker | TruLens | Weights and Biases
Education
Bachelor of Engineering | Bachelor of Science | Master of Science
Roles
AI | AI Engineering | AI Engineering Manager | AI Test Engineer | Engineer | Engineering Manager | Manager | Test Engineer
Related jobs
-
Featured Feat. Principal Knowledge & Data Architect USD 174K-284KAWS Neptune | Canonicalization | Chunking | Cypher | DBTBenefits including health and wellness programs | Health, wellness, and retirement plansSenior-level Full TimeHeadquarters - Chevy Chase, MD R15d ago
-
Featured Feat. Associate Director, Data Labs USD 167K-167KAWS | Cloud Computing | Compute Infrastructure | Data Analysis | LLM GovernanceConference speaking opportunities | Hybrid work schedule | Media appearancesSenior-level Full TimeWashington, District of Columbia, 20004, United … R28d ago
-
Artificial Intelligence Business Systems Analyst USD 96K-192KAnomaly Detection | Artificial Intelligence | Automation Anywhere | Azure DevOps | BPMN401k employer match | Company Paid Holiday | Dental insurance | Disability insurance | Employee assistance programMid-level Full TimeKnightdale, NC, United States7h ago
-
Embedded Software Engineering Intern USD 64K-82KARM Cortex | ARM Cortex-M | Assembly | C# | CANOpportunity for growth and development | Overtime payEntry-level Full Time InternshipBaxter Aerospace (St. George UT)8h ago
-
AI Developer Fall Intern USD 60K-80KAPI Design | Algorithms | Backend Development | ChatGPT | Cloud ComputingEntry-level InternshipEllicott City, MD, United States9h ago
-
AUTOSAR | C++ | CAN | Embedded Software | Embedded SystemsSenior-level Full TimeAllen Park, United States10h ago
-
Staff Product Engineer USD 160K-240KAWS | DSP | Data Pipelines | FFmpeg | Generative AI401k access | Dental insurance | Flexible PTO | Health insurance | Paid mealsSenior-level Full TimeSan Francisco, CA, US10h ago
-
SOFTWARE DATA ENGINEER - AI/ML/LLM - 2+ yrs of Experience - TS/SCI w/Poly clearance is required - ES - 062526-4 A USD 142K-146KArtificial Intelligence | Data Engineering | Data Modeling | Language Models | Large Language Models401k | Dental insurance | Medical insurance | Paid time off | Vision insuranceMid-level Full TimeFort George G Meade, United States10h ago
-
Senior Software Engineer - Analytics Platform USD 160K-240KApache Arrow | Apache Kafka | Asynchronous frameworks | C plus plus | DockerSenior-level Full TimeNew York10h ago
-
AI/BI Analyst – Hybrid Role USD 74K-120KAWS | Amazon Bedrock | Data Governance | Data Modeling | DatabricksDental insurance | Educational assistance | Flexible spending account | Health savings account | Life insuranceMid-level Full TimeAtlanta, Georgia R11h ago
-
Senior-level Full TimeHuntsville, United States11h ago
-
Data Engineer Junior USD 81K-125KAccess Management | Apache Airflow | Apache Spark | Automated testing | Data GovernanceEntry-level Full TimeSan Antonio, TX, United States11h ago
-
Data Engineer Junior USD 81K-125KAccess Management | Anomaly Detection | Apache Airflow | Apache Spark | Automated testingEntry-level Full TimeFairfax, VA, United States11h ago
-
.NET | Agile | Amazon Web Services | Angular | C#Travel opportunitiesMid-level Full TimeGilbert, Arizona, United States; Lake Mary, …12h ago
-
Data Engineer II USD 97K-162KAWS | Access Management | Agile | Artificial Intelligence | Cloud NetworkingMid-level Full TimeJersey City, New Jersey, United States12h ago
-
Applied AI Engineer II - Encore Program USD 102K-188K.NET | AWS | AWS Bedrock | Angular | AnthropicHybrid work 3 days per week | Mentorship | Professional growth | Travel 10 percentMid-level Full TimeDallas, Texas, United States; Hermitage, Tennessee, …12h ago
-
.NET | AWS | Agile | Angular | AzureSenior-level Full TimeGilbert, Arizona, United States; Lake Mary, …12h ago
-
Product Policy Manager, GenAI USD 161K-228KAccuracy review | Adversarial Simulation | Artificial Intelligence | Bias Mitigation | ComplianceSenior-level Full TimeWashington, DC13h ago
-
Senior Agent Engineer, Google Cloud, Applied AI USD 174K-253KAudio generation | BigQuery | CI/CD | Cloud Storage | Computer VisionBenefits | Bonus | EquitySenior-level Full TimeSunnyvale, CA, USA; Kirkland, WA, USA13h ago
-
Agentic development | Artificial Intelligence | Build Automation | CI/CD | Cloud infrastructureMid-level Full TimeSunnyvale, CA, USA13h ago
-
Abuse detection | Algorithms | Artificial Intelligence | Authentication | AuthorizationSenior-level Full TimeKirkland, WA, USA13h ago
-
C++ | Debugging | GPU | Java | LLMSenior-level Full TimeNew York, NY, USA; Cambridge, MA, …13h ago
-
Senior Software Engineer, AI/ML, Google Ads USD 174K-253KC++ | Data Processing | Data Structures | Data structures algorithms | DebuggingSenior-level Full TimeMountain View, CA, USA13h ago
-
Senior Software Engineer, Database Internals, AlloyDB USD 174K-253KC# | C++ | Code optimization | Compute | Concurrency ControlEntry-level Full TimeSunnyvale, CA, USA; Kirkland, WA, USA13h ago
-
Staff Software Engineer, Agentic Data and Evals USD 207K-301KAlgorithms | C++ | CSS | Data Storage | Data StructuresSenior-level Full TimeSunnyvale, CA, USA13h ago