Manager, Content Engineering — AI Content Understanding
Menlo Park, CA | New York, NY | San Francisco, CA
USD 134K-196K (estimate) Mid-level Full Time
Tasks
- Build repeatable evaluation sprint processes
- Calibrate automated evaluations using human data
- Create and maintain golden datasets
- Define evaluation guidelines and quality rubrics
- Identify and mitigate operational risks in evaluation pipelines
- Manage and develop content engineering team
- Manage workforce planning and performance management
- Own continuous content understanding model evaluations
- Report evaluation results to cross functional leadership
- Train reviewers and run calibration sessions
- Translate evaluation insights into model and prompt improvements
Perks/Benefits
- N/A
Skills/Tech-stack
A/B | A/B Testing | Annotation | B testing | Content labeling | Data Analysis | Evaluation Frameworks | Golden datasets | LLM-as-a-Judge | Machine Learning | Model Benchmarking | Prompt engineering | Python | Quality Rubrics | Recommendation Systems | Red Teaming | SQL | Search & Ranking | Trust and Safety
Education
Regions
Countries
States
Related jobs
-
AI Developer II USD 96K-150KAPIs | Agentic AI | Authentication and Authorization | Azure AI | Blue PrismEntry-level Full TimeMaryville, TN, United States5h ago
-
Senior Applied AI Engineer - Life Sciences & Healthcare USD 120K-200KAPI | AWS | CI/CD | Caching | Data IngestionSenior-level Full TimeAustin, TX, US7h ago
-
Automated testing | BM25 | Benchmarking | Chunking | Code ExecutionSenior-level Full TimeSeattle, Washington, United States8h ago
-
Manager, Analytics USD 106K-120KData Analysis | Data Visualization | Project Management | Sales operations | Tableau401k | Medical/Dental/Vision insurance | Paid time offMid-level Full TimeGA, United States15h ago
-
AI Engineering Intern (Hybrid) USD 40K-52KAutomation | Data Analysis | JavaScript | LLM Agents | Language ModelsHybrid work schedule | Internship program | Paid internshipEntry-level InternshipSan Francisco R18h ago
-
Senior Solutions Architect, Generative AI USD 184K-287KC# | C++ | Deep learning | Distributed Computing | GPUComprehensive benefits | EquitySenior-level Full TimeUS, CA, Santa Clara, United States20h ago
-
Entry-level Full TimeAlpharetta - HQ, United States20h ago
-
Senior Data Science Manager USD 115K-217KAzure | Data Engineering | Data Modeling | Data Transformation | Data Validation401k savings plan | Caregiver support | Comprehensive healthcare coverage | Employee assistance program | Mental health supportSenior-level Full TimeRichmond, Virginia, United States20h ago
-
Data Manager USD 102K-141KData Analysis | Data Cataloging | Data Compliance | Data Curation | Data GovernanceMid-level Full TimeChantilly, VA1d ago
-
Amazon Web Services | Anthropic | Cloud platform | Cohere | Google CloudCross-functional collaboration | Mentoring | Team research discussionsSenior-level Full TimeRemote, US or Europe R1d ago
-
Artificial Intelligence | Cloud Computing | Competitive Analysis | Content development | Customer ROIMid-level Full TimeSan Jose, CA, United States1d ago
-
Mid-level Full TimeSan Francisco, CA1d ago
-
A/B | A/B Testing | AWS | Agile | Artificial IntelligenceMid-level Full TimeSan Francisco, CA1d ago
-
Agile | Business glossary | Collibra | Data Cataloging | Data GovernanceRemote work optionMid-level Full TimeDallas, TX1d ago
-
Apache Airflow | Apache Hive | Apache Spark | BigQuery | Cloud platformMid-level Full TimeSunnyvale, CA1d ago
-
Cloud platform | Data Modeling | Data analytics | Google Cloud | Google Cloud PlatformMid-level Full TimeSan Bruno, CA1d ago
-
AWS | Agile methodologies | Algorithms | Behavior-Driven Development | Data StructuresDental insurance | Gourmet coffee | Gym membership reimbursement | Health insurance | Quiet work environmentSenior-level Full TimeCambridge, MA1d ago
-
API Design | Agile | Asynchronous programming | Automated testing | Autonomous PlanningTeam collaborationMid-level Full TimeBoston, MA1d ago
-
API Integration | Agile | Amazon Web Services | Ansible | AuroraSenior-level Full TimeBoston, MA1d ago
-
Amazon Web Services | Apache Hadoop | Apache Hive | Apache Spark | Cloud platformMid-level Full TimeBoston, MA1d ago
-
A/B | A/B Testing | AB Tasty | B testing | Conversion RateMid-level Full TimeBoston, MA1d ago
-
Data Pipelines | Distributed Systems | MLOps | Machine Learning | Model DeploymentCompetitive benefits | Hybrid workMid-level Full TimeSunnyvale, California, United States1d ago
-
AI Research Engineer USD 152K-258KContrastive Learning | Deep learning | Distributed Computing | Fine Tuning | Generative AIDental insurance | Flexible-hybrid work | Health insurance | Relocation assistance | Retirement planMid-level Full TimePalo Alto, California, United States1d ago
-
Associate Director, AI Solutions USD 80K-105KAWS | AWS Bedrock | Azure | Azure OpenAI | CI/CDHybrid work | Professional developmentMid-level Full TimeUnited States1d ago
-
Director, AI Solutions & Integration USD 149K-228KAWS | Agentic Workflows | Agile | Alerting | Artificial IntelligenceExecutive-level Full TimeUS - NY NYC - 55 …1d ago