Senior Software Development Engineer in Test — LLM Evaluation & Automation, T3E
Tasks
- Assess generated summaries and insights quality
- Build evaluation infrastructure for regression detection
- Build model evaluation coverage end to end
- Create component level evaluation tooling
- Debug evaluation failures and triage regressions
- Design LLM evaluation automation
- Develop evaluation rubrics and scoring
- Evaluate natural language generation against intent
- Implement LLM as a judge pipelines
- Improve reliability and repeatability of eval scoring
- Integrate evaluation jobs into existing pipelines
- Replace string matching with LLM judge checks
- Report quality trends and insights
- Validate image visual generation quality and metadata
Perks/Benefits
- N/A
Skills/Tech-stack
Bias and fairness | CI/CD | Dashboards | Data Bias and Fairness | Data bias | Dataset Management | Debugging | Infrastructure as Code | JSON | Jupyter | LLM Evaluation | Python | QA | REST APIs | Rubric Design | SQL | Test automation | YAML | “as-code”
Education
Regions
Countries
States
Cities
Related jobs
-
Embedded Software Development Engineer (Senior Level) USD 120K-160KC# | C++ | Critical software | DO-178 | Embedded SystemsCritical illness insurance | Employee assistance program | Health insurance | Legal assistance | Mental health benefitsSenior-level Full TimeEast Hartford, CT, United States R1d ago
-
Quantum Systems Software Development Engineer II USD 102K-202KAPI Development | C# | C++ | CI/CD | Data acquisitionCollaborative team | Learning opportunitiesEntry-level Full TimeUnited States, Washington, Redmond5d ago
-
AI Software Development Engineer USD 100K-192KAUTOSAR | Bazel | C# | C++ | CI/CDHealth insurance | Paid time off | Paid volunteer hours | Parental leave | Retirement savings planMid-level Full TimeAuburn Hills, MI, United States5d ago
-
Software Development Engineer 2 USD 96K-115KArchitecture Decision | Architecture Decision Records | CI/CD | DBT | Data ModelingDental insurance | Disability insurance | Flexible spending account | Health insurance | Health savings accountMid-level Full TimeGlobal Headquarters, United States6d ago
-
AI/LLM Network Software Development Engineer Graduate (High Speed Network) - 2027 Start (PhD) USD 162K-316KAI network | AI network optimization | C# | C++ | Computer Networking401k with company match | Dental insurance | Life insurance | Long-term disability | Medical insuranceEntry-level Full TimeSan Jose, California, United States6d ago
-
Alerting | Automated Deployment | Backup and Restore | ClickHouse | ConsulSenior-level Full TimeSeattle, Washington, United States6d ago
-
AI/LLM Network Software Development Engineer USD 207K-368KC# | C++ | Computer Networking | Congestion Control | Go401k match | Dental insurance | Health insurance | Life insurance | Long-term disabilityMid-level Full TimeSeattle, Washington, United States6d ago
-
AI/LLM Network Software Development Engineer Graduate (High Speed Network) - 2026 Start (PhD) USD 162K-316KAI network | AI network optimization | C++ | Computer Networking | Congestion Control401k savings plan | Life insurance | Long-term disability | Medical, dental, and vision insurance | Paid HolidaysEntry-level Full TimeSan Jose, California, United States6d ago
-
AI/LLM Network Software Development Engineer - San Jose USD 218K-480KAI network | AI network optimization | C++ | Computer Networking | Congestion Control401k match | Dental insurance | Life insurance | Long-term disability | Medical insuranceMid-level Full TimeSan Jose, California, United States6d ago
-
AI/LLM Network Software Development Engineer Graduate (High Speed Network) - 2026 Start (PhD) USD 153K-300KAI network | AI network optimization | C++ | Computer networks | Congestion Control401k match | Life insurance | Long-Term Disability coverage | Medical/Dental/Vision insurance | Paid HolidaysEntry-level Full TimeSeattle, Washington, United States6d ago
-
AI/LLM Network Software Development Engineer Graduate (High Speed Network) - 2027 Start (PhD) USD 153K-300KAI network | AI network optimization | C# | CPP | Computer Networking401k match | Life insurance | Long-term disability | Medical/Dental/Vision insurance | Paid HolidaysEntry-level Full TimeSeattle, Washington, United States6d ago
-
API Automation | Agile | Confluence | Data Migration | Data QualitySenior-level Full TimeSouthlake, TX, United States8d ago
-
AOT Tracing | Build systems | C++ | CUDA | CUDA kernelsCareer growth | Code review feedback | Continuous learning | Cross-functional collaboration | MentorshipSenior-level Full TimeCupertino, California, USA8d ago
-
API Integration | Artificial Intelligence | Iterative design | JavaScript | PythonEntry-level InternshipBentonville, AR8d ago
-
ADC | Bare Metal | Bring-up | C# | C++Senior-level Full TimeSunnyvale, CA, United States12d ago
-
Bare Metal | C++ | Embedded C | Emulation | Linux401k matching | Health insurance | Paid time off | Parental leaveMid-level Full TimeAustin, Texas, USA13d ago
-
Senior Software Development Engineer - Iris AI USD 130K-162K.NET | API Design | Amazon Bedrock | Anthropic API | Asynchronous ArchitectureEmployee assistance program | Health insurance | Life insurance | Paid Company Holidays | Paid time offSenior-level Full TimeU.S. Remote R13d ago
-
AWS Athena | Amazon S3 | Apache Iceberg | Asynchronous Architecture | Build Automation401k matching | Flexible work arrangements | Health insurance | Paid time off | Parental leaveMid-level Full TimeArlington, Virginia, USA14d ago
-
Software Development Engineer, AWS Bill Generation USD 158K-213KAWS | Agile | Amazon Web Services | C# | C++401k matching | Dental insurance | Health insurance | Paid time off | Parental leaveMid-level Full TimeNew York, New York, USA15d ago
-
Computer Scientist USD 164K-238KAndroid | App Development | Artificial Intelligence | C++ | Camera App DevelopmentEntry-level Full TimeSan Jose, United States R15d ago
-
Lead Engineer IT Software Development & Maintenance USD 160K-211KAI Agents | AMQ | API Gateway | AWS | AWS LambdaHybrid work | Limited travelSenior-level Full TimeGA, United States R15d ago
-
.NET | Azure Pipelines | C# | CI/CD | Computer Engineering401k plan | Dependent care support | Disability programs | Education assistance | Generous parental leaveSenior-level Full TimeUS-NY-SKANEATELES-721 VISIONS DRIVE, United States R19d ago
-
2882 Software Engineer 3 USD 50KAlgorithm Design | Database Design | Debugging | Documentation | Performance Engineering401k match | Dental insurance | Healthcare | Life insurance | Paid time offSenior-level Full TimeMaryland, Fort Meade, United States of …19d ago
-
C++ | DNS | Dependency Injection | Firewalls | HTTP/3Education reimbursement | Health plans | Hybrid work model | Parental leave options | Retirement optionsSenior-level Full TimeBellevue, Washington, USA; San Jose, California, … R20d ago
-
Software Development Engineer II, AWS SageMaker AI USD 143K-194KAWS | AWS Batch | AWS CDK | AWS Step Functions | Airflow401k matching | Employee assistance program | Health insurance | Paid time off | Parental leaveMid-level Full TimeBellevue, Washington, USA21d ago