Product Manager - AI Evaluation & Developer Platforms
São Paulo, State of São Paulo, Brazil
A BRL 271K-370K (estimate) Senior-level Full Time
Tasks
- Build evaluation and developer tooling systems
- Collaborate with engineering on delivery cycles
- Create benchmarks and scoring methodologies
- Define AI quality standards
- Define and monitor KPIs and OKRs
- Deliver evaluation dashboards and developer experiences
- Develop human feedback annotation workflows
- Enable dataset creation experiment execution and results analysis
- Ensure traceability from requirements to delivered capabilities
- Own AI evaluation roadmap
- Present findings to stakeholders
- Track tool adoption effectiveness and business impact
- Write product requirements and acceptance criteria
Perks/Benefits
- N/A
Skills/Tech-stack
AI Agents | AI Quality Assurance | AI quality | API Integration | Agile | Annotation | Artificial Intelligence | Benchmarking | Data Governance | Data Ingestion | Data Quality | Data quality monitoring | ETL | Evaluation methodology | Human Feedback | Knowledge of Metrics | LLM | Language Models | Large Language Models | Machine Learning | Product Management | Quality Assurance | Quality monitoring | RAG | Retrieval-Augmented Generation | Test strategy
Education
N/A
Roles
Related jobs
-
AI Engineer USD 125K-197KAgentic Architectures | Langchain | Language Models | Large Language Models | Model OrchestrationMid-level Full TimeCanada1d ago
-
Forward Deployed Engineer, Enterprise AI [LATAM] BRL 319K-396KAWS | Automated testing | Cloud Computing | Cloud platform | ComplianceSenior-level Full TimeSão Paulo, Brazil3d ago
-
Adversarial Testing | Artificial Intelligence | Bias Assessment | Experiment tracking | Hallucination detectionContract based opportunity | Opportunity to work on advanced AI systems | Remote work flexibilitySenior-level Contract Full TimeBrazil R3d ago
-
Arquitetura de Dados e IA | Remota - 134857 BRL 119K-119KAWS Glue | AWS SageMaker | Agile | Amazon Athena | Amazon EMRRemote workSenior-level Full TimeAlphaville - Barueri, BR, 06.454-0003d ago
-
AI Governance | AWS | Agent systems | Azure | Cloud ArchitectureContract extension opportunity | Flexible consulting environment | Fully remote | US time zone overlapSenior-level Contract Full TimeBrazil R4d ago
-
AI Agents | AWS Bedrock | Anthropic Claude | Cause analysis | Cloud infrastructureChildcare assistance | Continuous learning opportunities | Dental insurance | Discount programs | Food allowanceSenior-level Full TimeBrazil4d ago
-
Apache Airflow | Code Refactoring | Data Visualization | Data layers | Data orchestrationDental insurance | Fully remote work | Health insurance | Learning incentives | Meal and food allowanceSenior-level Full TimeBrazil R4d ago
-
A/B | A/B Testing | APIs | Analytics | B testingEquity opportunities | Flexible time off | Flexible work hours | Fully remote work | Learning and development budgetSenior-level Full TimeBrazil R4d ago
-
Data Engineer Manager BRL 13K-18KAzure | Cloud Architecture | Data Architecture | Data Engineering | Data GovernanceChildcare support | Corporate university | Dental insurance | Educational partnerships | Health insuranceMid-level Full TimeOSASCO, Brazil; CURITIBA, PR, Brazil; OSASCO, …4d ago
-
Artificial Intelligence | Artificial Intelligence Reasoning | Attention to detail | Content Moderation | Data AnnotationFreelance project | Part-time | Remote workEntry-level FreelanceBrazil - Remote R4d ago
-
AI Engineering I - Marketing USD 30K-39KAWS | BERT | CI/CD | Docker | EmbeddingsFull-time employment | Remote workEntry-level Full TimeBrazil, Remote R4d ago
-
Analytical modeling | B2B | Cloud | Credit Risk | Data LakeCareer development | Health and wellness benefits | Hybrid workMid-level Full TimeSão Paulo, Brazil4d ago
-
Access Control | Data Governance | Data Modeling | Python | SQLChildcare assistance | Continuous learning opportunities | Dental insurance | Discount club partnerships | Extended paternity leaveMid-level Full TimeBrazil5d ago
-
.NET | AI Search | AWS | Agent systems | AgileRemote workSenior-level Full TimeAlphaville - Barueri, BR, 06.454-0005d ago
-
AI Engineer II - Marketing USD 39K-72KAWS | CI/CD | Docker | Entity recognition | Feature EngineeringFull-time employment | Remote workMid-level Full TimeBrazil, Remote R5d ago
-
AI Agents | API Integration | AWS | Autogen | AzureSenior-level Full TimeSão Paulo, Brazil5d ago
-
AI Engineer II (Remote) USD 39K-72KAWS | BERT | CI/CD | Docker | EmbeddingsFull-time employment | Remote workMid-level Full TimeBrazil, Remote R5d ago
-
[Job-30263] AI Data Engineer / Snowflake Cortex Engineer BRL 180K-180KAccess Control | Anomaly Detection | Data Engineering | Data Governance | Data ModelingChildcare assistance | Continuous learning platform | Discount club | Extended paternity leave | Free well being platformSenior-level Full TimeBrazil5d ago
-
AI Agents | AWS Lambda | Amazon S3 | Amazon SQS | Automated testingChildcare assistance | Continuous learning | Dental insurance | Discount club | Extended parental leaveSenior-level Full TimeBrazil6d ago
-
AI Deployment Engineer, Enterprise BRL 300K-360KCost Optimization | Data Governance | Debugging | Evaluation | GovernanceSenior-level Full TimeSão Paulo6d ago
-
API Development | Autogen | CrewAI | Data Ingestion | Data ModelingEquipment support | Flexible working hours | Gifts and rewards | Lunch allowance | No dress codeSenior-level Full TimeBrazil R7d ago
-
AMP | Asynchronous programming | CI/CD | Claude Code | Cloudflare VectorizeHybrid work | Medical healthcare | Ongoing learning reimbursement | Recognition program | Referral bonusesSenior-level Full TimeBrazil7d ago
-
Gerente de Data Science BRL 21K-24KARIMA | Artificial neural networks | C# | C++ | ClusteringHybrid work modelMid-level Full TimeBusiness Unit Head Office - São … R7d ago
-
Sr Specialist, AI Platform Engineering BRL 235K-390KAWS | Airflow | CI/CD | Distributed Systems | DockerLife insurance | Meal vouchers | Medical and dental assistance | Others | Private pensionSenior-level Full TimeJundiai, SP, BR7d ago
-
Sr Specialist, AI Platform Engineering BRL 235K-390KAirflow | Amazon Web Services | CI/CD | Cloud platform | Distributed SystemsDental insurance | Life insurance | Meal vouchers | Medical insurance | Private pensionSenior-level Full TimeJundiai, SP, BR7d ago