Senior Software Development Engineer in Test — LLM Evaluation & Automation, T3E
Tasks
- Assess generated summaries and insights quality
- Build evaluation infrastructure for regression detection
- Build model evaluation coverage end to end
- Create component level evaluation tooling
- Debug evaluation failures and triage regressions
- Design LLM evaluation automation
- Develop evaluation rubrics and scoring
- Evaluate natural language generation against intent
- Implement LLM as a judge pipelines
- Improve reliability and repeatability of eval scoring
- Integrate evaluation jobs into existing pipelines
- Replace string matching with LLM judge checks
- Report quality trends and insights
- Validate image visual generation quality and metadata
Perks/Benefits
- N/A
Skills/Tech-stack
Bias and fairness | CI/CD | Dashboards | Data Bias and Fairness | Data bias | Dataset Management | Debugging | Infrastructure as Code | JSON | Jupyter | LLM Evaluation | Python | QA | REST APIs | Rubric Design | SQL | Test automation | YAML | “as-code”
Education
Regions
Countries
States
Cities
Related jobs
-
Senior-level Full TimeLexington, MA; Boston, MA; Boston, Massachusetts; …9h ago
-
Software Development Engineer - Cloud Native Databases USD 148K-300KBlob Storage | Database Schema | Database schema management | Distributed Systems | Distributed TransactionsAgile collaborative environment | On-call rotationMid-level Full TimeSeattle, Washington, United States13h ago
-
Alerting | Backup and Restore | Cloud infrastructure | Disaster Recovery | Distributed SystemsEntry-level Full TimeSeattle, Washington, United States13h ago
-
Senior-level Full TimeHome - United States National (HOMEUSN)1d ago
-
Executive-level Full TimeLocation Irvine, California2d ago
-
.NET | Azure Pipelines | C# | Computer Vision | Continuous integrationDependent care support | Disability programs | Education assistance | Generous parental leave | Leadership developmentSenior-level Full TimeUS-NY-SKANEATELES-721 VISIONS DRIVE, United States5d ago
-
Software Development Manager, PXTCS (People Experience and Technology Central Science) USD 184K-250KAlerting | Artificial Intelligence | Build Automation | CI/CD | Cause analysis401k matching | Dental insurance | Employee assistance program | Flexible spending accounts | Health insuranceMid-level Full TimeBellevue, Washington, USA5d ago
-
CI/CD | Confluence | DAX | Docker | GitLabHybrid work | Occasional travelSenior-level Full TimeAtlanta, GA, US R6d ago
-
Software Development Engineer, Amazon MSK USD 168K-227KAmazon Kinesis | Apache Flink | Apache Kafka | Apache Spark | Apache Storm401k matching | Employee mentorship | Health insurance | Paid time off | Parental leaveSenior-level Full TimeSanta Monica, California, USA7d ago
-
Software Development Engineer, Amazon MSK USD 143K-194KAmazon Kinesis | Apache Flink | Apache Kafka | Apache Spark | Apache StormCareer growth | Flexible work schedule | Mentorship | Paid time off | Parental leaveMid-level Full TimeSanta Monica, California, USA7d ago
-
Software Development Engineer - Cloud Native Databases USD 156K-316KBackup and Restore | Blob Storage | Crash diagnostics | Database Schema | Database schema managementCollaborative work environment | Intellectual curiosity | On-call rotationMid-level Full TimeSan Jose, California, United States7d ago
-
Agile | Data Architecture | Data Governance | Data Integrity | Data Modeling401k | Employee assistance program | Employee discounts | Health, dental, and vision plans | Life insuranceSenior-level Full TimeArlington, VA, Remote, United States R8d ago
-
AI Safety | AWS | Agentic AI | Azure | CI/CDDental insurance | Medical insurance | Paid time off | Retirement savings options | Vision insuranceSenior-level Full TimeRichardson-909 E Collins Blvd, United States8d ago
-
Software Development Engineer, Amazon Fulfillment Technology (AFT), Platform Engineering and Services (AFT Rodeo) USD 143K-194KAPI Development | AWS | CloudWatch | Data Pipelines | Design Patterns401k matching | Adoption reimbursement | Dental insurance | Flexible spending accounts | Health insuranceMid-level Full TimeBellevue, Washington, USA8d ago
-
Software Development Engineer, AWS Agentic AI USD 143K-194KAPIs | Build processes | C# | C++ | CLI401k matching | Adoption and surrogacy reimbursement | Employee assistance program | Flexible spending accounts | Health insuranceMid-level Full TimeSeattle, Washington, USA8d ago
-
Amazon Trainium | Automation | Collectives | Design Patterns | Distributed inference401k matching | Employee assistance program | Flexible spending accounts | Health insurance | Mental health supportMid-level Full TimeCupertino, California, USA14d ago
-
Software Development Engineer in Test, Machine Learning USD 155K-220KAWS | Agentic Systems | CI/CD | Dashboards | Data VisualizationMid-level Full TimeFoster City, CA18d ago
-
Software Dev Architect I, Data & Analytics USD 150K-200KAPI Design | AWS | Angular | Automated testing | AzureSenior-level Full TimeTulsa Main Office, United States22d ago
-
Software Dev Engineer, Applied AI USD 143K-194KAmazon SageMaker | Build processes | Design Patterns | Distributed Systems | Docker401k matching | Paid time off | Parental leaveMid-level Full TimeBellevue, Washington, USA22d ago
-
Software Development Engineer, Benchmarks USD 158K-213KAWS | Alerting | Apache Hadoop | Apache Iceberg | Apache Spark401k matching | Dental insurance | EAP | Flexible spending accounts | Health insuranceMid-level Full TimeNew York, New York, USA22d ago
-
C# | C++ | CI/CD | Git | Hardware-in-the-loopEntry-level Full TimeAuburn Hills, MI, United States29d ago
-
AWS | Algorithms | Big Data | C# | C++401k matching | Adoption and surrogacy reimbursement | Dental insurance | Employee assistance program | Flexible spending accountsSenior-level Full TimeSeattle, Washington, USA1mo ago
-
Access Management | CI/CD | Distributed Systems | Docker | Git401k matching | EAP | Flexible spending accounts | Health insurance | Paid time offSenior-level Full TimeSeattle, Washington, USA1mo ago
-
Algorithms | C# | C++ | CI/CD | Data Structures401k matching | Adoption and surrogacy reimbursement | Dental insurance | EAP | Flexible spending accountsSenior-level Full TimeCupertino, California, USA1mo ago
-
Embedded SDE II, Prime Air USD 143K-194KBoard Bring-up | Bring-up | Build process | C# | C++401k matching | EAP | Flexible spending accounts | Health insurance | Mental health supportMid-level Full TimeSeattle, Washington, USA1mo ago