QA Engineer - Gen AI
Austin, Texas, United States
USD 135K-260K (estimate) Senior-level Full Time
Tasks
- Apply prompt engineering testing best practices
- Automate LLM performance monitoring
- Build tools and dashboards for LLM quality
- Calibrate automated scoring
- Collaborate with ML engineers and product teams
- Curate versioned ground truth datasets
- Design data quality checks
- Design regression test suites for LLM evaluation
- Evaluate structured extraction against ground truth
- Identify LLM failure modes
- Validate model output consistency
Perks/Benefits
- N/A
Skills/Tech-stack
AWS | AWS SageMaker | Amazon EKS | Amazon S3 | Anthropic | CI/CD | Data Quality | Data Validation | Databricks | Datadog | DeepEval | Document processing | F1-score | Fuzzy Matching | Human-in-the-loop | Hypothesis | Information Extraction | Kubernetes | Langsmith | Language Models | Language Processing | Large Language Models | MLOps | MLflow | Machine Learning | Model Evaluation | Natural Language | Natural Language Processing | Numeric Tolerance | OCR | Observability | OpenAI | PDF Processing | PostgreSQL | Precisión | Prompt engineering | Pytest | Python | RAG | Ragas | Recall | Regression testing | SQL | Structured Information Extraction | Table Extraction | The Loop
Education
N/A
Roles
Related jobs
-
Featured Feat. Principal Knowledge & Data Architect USD 174K-284KAWS Neptune | Canonicalization | Chunking | Cypher | DBTBenefits including health and wellness programs | Health, wellness, and retirement plansSenior-level Full TimeHeadquarters - Chevy Chase, MD R26d ago
-
PIIM Data Engineer USD 105K-160KAPIs | Azure Databricks | Data Governance | Data Lake | Data ModelingCareer growth | Relocation assistance | Time to rechargeMid-level Full TimeNew Albany, OH, US6h ago
-
Senior/Staff Data Scientist USD 184K-200KConvolutional Neural Networks | Data Preprocessing | Data cleaning | Deep learning | Evaluation metrics401k match | Cell phone stipend | Company Wellness Days | Dental insurance | Employee stock purchase planSenior-level Full TimeSunnyvale, CA6h ago
-
Design Reliability Engineer (Falcon & Dragon) USD 115K-160KABAQUS | ANSYS | Automation | Composite layup | Data Analysis401k retirement plan | Dental coverage | Employee stock purchase plan | Medical coverage | Paid HolidaysEntry-level Full TimeHawthorne, CA7h ago
-
Software Engineer, Embedded UI USD 170K-210KAI Agents | API Design | Automation | BrightScript | C++Disability benefits | Equity awards | Financial wellness support | Flexible work schedule | Health insuranceSenior-level Full TimeSan Jose, California7h ago
-
AI Search | Application Insights | Azure AI | Azure AI Search | Azure Container401k | Commuter benefits | Dental insurance | Educational assistance | Health savings accountMid-level Full TimeNew York, United States R7h ago
-
Robotics AI Engineer - Sensor Calibration USD 70K-300K3D Geometry | C++ | Camera Calibration | Ceres-Solver | Computer VisionSenior-level Full TimeIrvine, CA8h ago
-
Staff Machine Learning Engineer - Ads USD 164K-282KAirflow | Apache Spark | BigQuery | Cloud Computing | Deep learning401k plan with company match | Flexible spending accounts | Health savings account | Medical/dental/vision/life insurance | Paid parental leaveSenior-level Full TimeMountain View, USA9h ago
-
Staff Machine Learning Engineer - Ads USD 164K-282KAirflow | Amazon SageMaker | Apache Spark | BigQuery | Contrastive Learning401k plan with company match | Electric Car Charging Station | Employee assistance program | Flexible spending accounts | Health savings accountSenior-level Full TimeMountain View, USA9h ago
-
AI Infrastructure Engineer – Agents & ML Systems USD 175K-200KAirflow | Amazon S3 | Backend Development | C++ | Dagster401k matching | Equity package | Global Entry | Health and wellness stipend | Health, dental, vision insuranceMid-level Full TimeRemote R10h ago
-
Staff Software Engineer, Storage USD 207K-303KAmazon S3 | C++ | Ceph | ClickHouse | DAOS401k match | Disability insurance | Employee stock purchase plan | Flexible PTO | Flexible spending accountSenior-level Full TimeWashington11h ago
-
Staff Software Engineer, Storage USD 207K-303KC++ | Ceph | ClickHouse | Distributed file systems | File systems401k match | Catered lunch | Childcare support | Dental insurance | Disability insuranceSenior-level Full TimeCalifornia11h ago
-
Software Development Engineer II, AWS SageMaker AI USD 143K-194KAWS Batch | AWS CDK | AWS CloudFormation | AWS SageMaker | AWS Step FunctionsMid-level Full TimeBellevue, Washington, USA11h ago
-
Senior Data Engineer USD 60K-76KAWS Glue | AWS Lambda | AWS S3 | AWS Secrets | AWS Secrets ManagerLearning and development resourcesSenior-level Full TimeNew York, US11h ago
-
Machine Learning Engineer - Voice Conversion USD 200K-220KAdversarial Training | Audio CODEC | Audio Variational Autoencoder | Bias Testing | CUDA401k retirement plan | Company equity | Fertility support | Lifestyle spending account | Meals and snacksSenior-level Full TimeRemote (U.S. or Europe) R11h ago
-
C++ | Causal reasoning | Deep learning | Foundation Models | Generative ModelsMid-level Full TimeMountain View, CA USA; San Francisco, …12h ago
-
Senior-level Full TimeNew York, NY - Hybrid (3x … R12h ago
-
Senior-level Full TimeDallas, TX - Hybrid (3x in … R12h ago
-
Senior-level Full TimeMcLean, VA; McLean, VIRGINIA; McLean, VIRGINIA; …13h ago
-
Senior-level Full TimeMcLean, VIRGINIA13h ago
-
Data Engineer (Expert) WA07 USD 133K-238KData Mapping | Data Modeling | Data Quality | Data Testing | ETLSenior-level Full TimeChantilly, Virginia13h ago
-
Data Engineer (Subject Matter Expert) WA28 USD 133K-238KApache NiFi | CSS | Data Modeling | Data Quality | ETLSenior-level Full TimeChantilly, VA13h ago
-
Data Engineer (Expert) WA12 USD 133K-238KApache NiFi | Data Mapping | Data Modeling | Data Quality | Data TestingSenior-level Full TimeMcLean, VIRGINIA13h ago
-
Data Engineer (Subject Matter Expert) CK16 USD 133K-238KApache NiFi | C# | C++ | Cloud Computing | Data MappingCompetitive benefits packageSenior-level Full TimeMcLean, VIRGINIA13h ago
-
System Engineer- Storage (Expert) TI28 USD 100K-150KActive Directory | Auditing | Backup and Recovery | Bash | Capacity PlanningSenior-level Full TimeMcLean, VIRGINIA13h ago