Manager, Content Engineering — AI Content Understanding
Menlo Park, CA | New York, NY | San Francisco, CA
USD 134K-196K (estimate) Mid-level Full Time
Tasks
- Build repeatable evaluation sprint processes
- Calibrate automated evaluations using human data
- Create and maintain golden datasets
- Define evaluation guidelines and quality rubrics
- Identify and mitigate operational risks in evaluation pipelines
- Manage and develop content engineering team
- Manage workforce planning and performance management
- Own continuous content understanding model evaluations
- Report evaluation results to cross functional leadership
- Train reviewers and run calibration sessions
- Translate evaluation insights into model and prompt improvements
Perks/Benefits
- N/A
Skills/Tech-stack
A/B | A/B Testing | Annotation | B testing | Content labeling | Data Analysis | Evaluation Frameworks | Golden datasets | LLM-as-a-Judge | Machine Learning | Model Benchmarking | Prompt engineering | Python | Quality Rubrics | Recommendation Systems | Red Teaming | SQL | Search & Ranking | Trust and Safety
Education
Regions
Countries
States
Related jobs
-
Manager, Marketplace Partner Analytics USD 110K-115KCapacity Planning | Data analytics | Demand Planning | Ecommerce | Excel401k retirement program | Dental insurance | Disability insurance | Employee assistance program | Employee discountMid-level Full TimeJericho, NY, United States8h ago
-
Director, AI Solutions Architect USD 139K-225KAPI Gateway | Agent Orchestration | BigQuery | CI/CD | Cloud MicroservicesAfter-hours support | Occasional overtime | Occasional travelSenior-level Full TimeOak Brook, IL, United States9h ago
-
Technical Architect – AI, ML & Generative AI USD 142K-240KAWS Bedrock | AWS SageMaker | Agentic AI | Apache Spark | Artificial Intelligence401k | Critical Illness Accident Hospital Indemnity Identity Theft Protection | Dental plans | Life and Accidental Death and Dismemberment | Long-term disabilitySenior-level Full TimeFrisco, United States12h ago
-
Classification Algorithms | Data Analysis | Deep learning | Language Models | Language ProcessingSenior-level Full TimeSan Jose, California, United States13h ago
-
Entry-level InternshipChicago, IL, US16h ago
-
Solution Architect (AI & Data Applications) USD 180K-247KAutogen | CI/CD | Databricks | Docker | FastAPIMentoring system | Professional development | Supportive work environmentSenior-level Full TimeJersey City, NJ, United States16h ago
-
Associate Director, Integrated Analytics - Asundexian USD 143K-214KAI | Claims data | Claims data analysis | Competitive Analysis | Data AnalysisDental insurance | Health insurance | Paid time off | Relocation assistance | Retirement planMid-level Full TimeWhippany, New Jersey, US17h ago
-
Attribution Modeling | Causal Inference | Customer Lifetime Value | Customer Segmentation | Difference-in-differences401k match | Annual bonus | Company equipment provided | Company subsidized medical dental and vision | Disability benefitsSenior-level Full TimeAtlanta, GA preferred, Remote R23h ago
-
AWS | Artificial Intelligence | Azure AI | Data Analysis | DatabricksBackup childcare | Financial coaching | Health care coverage | Mental health support | On-site health and wellness centersSenior-level Full TimeChicago, IL, United States1d ago
-
Advanced Analytics Manager USD 117K-209KAzure | Business Intelligence | Dashboarding | Data Governance | Data ModelingMid-level Full TimeCambridge - B3 Crossing, United States1d ago
-
Senior Supplier Fulfillment Leader USD 116K-155KChain management | Contract Negotiation | Data Analysis | Enterprise Resource Planning | Inventory ManagementDental insurance | Disability insurance | Employee assistance program | Health coaching | Health insuranceSenior-level Full TimeRemote, United States R1d ago
-
Associate Director, Computational Biology USD 180K-240KAgent-based | Agent-based modeling | Bioinformatics Databases | Cell biology | D3JS401k | Dental insurance | ESPP | Employee wellness | Medical insuranceMid-level Full TimeSilver Spring, MD, United States1d ago
-
Manager, Data Science USD 60K-75KAttribution | Data Mining | Data Preparation | Data Transformation | Explainable AIHybrid work model | Visa sponsorshipMid-level Full TimeMalvern, PA, United States1d ago
-
Manager, Data Science USD 91K-118KData Products | Elasticsearch | Experimentation | Language Processing | Machine LearningHealth benefits | Wellness benefitsMid-level Full TimeDetroit - One Campus Martius, United …1d ago
-
BigQuery | Cloud Computing | Dashboards | Data Engineering | Data GovernanceDental coverage | Medical coverage | Paid time off | Retirement savings options | Vision coverageSenior-level Full TimeWork At Home-Rhode Island, United States1d ago
-
Senior Manager, Analytics & Insights USD 82K-182KData Modeling | Data Visualization | Data Warehousing | Python | Query OptimizationHybrid work environment | Medical, dental, and vision coverage | Paid time off | Retirement savings options | Wellness programsSenior-level Full TimeHartford-Farmington Ave Atrium, United States1d ago
-
Mid-level ContractHarrisburg, PA1d ago
-
AI Engineering Sr Director or VP, Data Science USD 128K-175KAI Platform | AWS SageMaker | Agent systems | Agentic AI | Azure MLCollaborative culture | Growth opportunities | Impactful technical work | Professional developmentSenior-level Full TimeColumbia, MD, United States1d ago
-
Senior Manager of Software Engineering for Data Platform USD 175K-185KAWS | Access Control | Alert Suppression | Alert escalation | Alert routingSenior-level Full TimeJersey City, NJ, United States1d ago
-
Senior Cybersecurity Analytics Manager USD 116K-184KBig Data | Cloud Computing | Cybersecurity monitoring | Data Transformation | Data analyticsFederal holidays off | Flexible PTO | Professional development support | Tuition reimbursement | Wellness stipendsSenior-level Full TimeWashington, D.C. Metro1d ago
-
AI Engineer USD 139K-198KAI Search | AKS | AWS Bedrock | Amazon SageMaker | AutogenLeadership development | Professional developmentSenior-level Full TimeWashington, DC1d ago
-
AI Agents | Apache Spark | Data Ingestion | Data Modeling | Data Transformation401k match | Company provided disability insurance | Dental insurance | Flexible spending accounts | Health care and dependent care flexible spending accountsSenior-level Full TimeUnited States1d ago
-
AI Solutions Engineer, East USD 125K-175KAWS | Azure | Cloud platform | Dspy | Generative AI401k plan | Dental insurance | Medical insurance | Mental wellness support | Parental leaveMid-level Full TimeRemote (New York) R1d ago
-
Data Science Manager - Sports Pricing USD 160K-190KMachine Learning | Optimization | Python | R | Risk Management401k with company match | Company equipment provided | Company in person events | Company subsidized medical dental vision | Disability benefitsMid-level Full TimeAtlanta, GA preferred, Remote R1d ago
-
Sr. AI Engineer USD 176K-240KAWS | Agentic Workflows | Autonomous Agents | Compliance | Context engineering401k plan with employer matching | Advancement opportunities | Employee development program stipend | Fertility/adoption assistance | Flexible PTOSenior-level Full TimeAtlanta, GA1d ago