AI Automation QA
Tasks
- Build automated evaluation harnesses for agent performance
- Collaborate with developers to improve prompts tools and workflows
- Continuously evaluate and monitor system behavior in production
- Define and execute quality assurance strategies for AI workflows
- Define quality criteria for agent workflows
- Develop dashboards and reports for quality metrics
- Develop runbooks for common failure modes
- Ensure testing logging and monitoring meet privacy and regulatory requirements
- Evaluate outputs using LLM evaluation frameworks
- Implement automated test suites using Playwright
Perks/Benefits
- N/A
Skills/Tech-stack
API Testing | API validation | Anomaly Detection | Data Analysis | DeepEval | Evidently | LLM Evaluation | Playwright | Production Monitoring | Python | Ragas | Regression testing | Test automation | UI Testing | Workflow testing
Education
N/A
Related jobs
-
Practice Customer Engineer, Data Analytics, Google Cloud MXN 900K-1100KApache Spark | Big Data | Data Modeling | Data Warehousing | DataprocSenior-level Full TimeMexico City, CDMX, Mexico1h ago
-
Analytics Senior Executive, GOC MXN 650K-815KAdobe Analytics | Campaign Manager | Cause analysis | Data Governance | Data ModelingEmployee discounts | Life insurance | Mental health support | Paid time off | Wellness programsSenior-level Full TimeBarranquilla, Colombia; Bogota, Colombia; Medellin, Colombia; …11h ago
-
Embedded SW Engineer - Qualcomm Platform MXN 100K-120KAOSP | C# | C++ | Case design | Code AnalysisEntry-level Full TimeGuadalajara, Jal., Mexico18h ago
-
Integration Engineer - Customer Data Platform (CDP) MXN 900K-1080KAPIs | Authentication | Cloud Data | Cloud Data Platforms | Data MatchingCompany-provided equipment | English lessons | Flexible working options | Learning opportunities | Mentorship programsMid-level Full TimeGuadalajara, Jal., Mexico21h ago
-
Senior Data Engineer ID71670 MXN 900K-1100KAWS | AWS Kinesis | Apache Airflow | Apache Kafka | Automated testingEducation budget | Exciting projects | Fitness budget | Flexible schedule | MentorshipSenior-level Full TimePuebla (Heroica Puebla), Mexico22h ago
-
Senior Data Engineer ID71670 MXN 900K-1100KAWS Kinesis | Apache Airflow | Apache Kafka | Automated testing | CI/CDEducation budget | Fitness budget | Flexible schedule | Mentorship | Office workSenior-level Full TimeGuadalajara, Mexico22h ago
-
Senior Data Engineer ID71670 MXN 900K-1100KAWS Kinesis | Apache Airflow | Apache Kafka | Automated testing | CI/CDFlextime | Mentorship | Personalized growth roadmaps | Remote and office options | TechtalksSenior-level Full TimeLeón de los Aldama, Mexico22h ago
-
Senior Data Engineer ID71670 MXN 900K-1100KAWS Kinesis | Apache Airflow | Apache Kafka | Batch Processing | CI/CDEducation budget | Fitness budget | Flexible schedule | Mentorship | Office optionsSenior-level Full TimeCiudad de México, Mexico22h ago
-
Senior Data Engineer ID71670 MXN 900K-1100KAWS Kinesis | Apache Airflow | Apache Kafka | Automated testing | Batch ProcessingEducation budget | Fitness budget | Flextime | Mentorship | Personalized growth roadmapSenior-level Full TimeMonterrey, Mexico22h ago
-
Senior Data Engineer ID71670 MXN 900K-1100KAWS Kinesis | Apache Airflow | Apache Kafka | CI/CD | DagsterEducation budget | Fitness budget | Flexible schedule | Mentorship | Office optionsSenior-level Full TimePlayas de Rosarito, Mexico22h ago
-
Senior Data Engineer ID71670 MXN 900K-1100KApache Airflow | Automated testing | Batch Processing | CI/CD | DagsterEducation budget | Fitness budget | Flexible schedule | Mentorship | Office work optionSenior-level Full TimeZapopan, Mexico22h ago
-
Senior Data Engineer ID71670 MXN 900K-1100KAWS Kinesis | Amazon Web Services | Apache Airflow | Apache Kafka | Automated testingFlexible schedule | Mentorship | Personalized growth roadmaps | Professional growth | Remote and office optionsSenior-level Full TimeQuerétaro, Mexico22h ago
-
Access Control | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake StorageCompany-sponsored certifications | Company-sponsored coursework | Continuing education | Flexible working hours | Paid time offSenior-level Full TimeZapopan, Jalisco, Mexico R1d ago
-
Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure DevOpsCompany-sponsored certifications | Conference attendance | Continuing education | Flexible working hours | Paid time offSenior-level Full TimeTijuana, Mexico R1d ago
-
Agile | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake StorageCompany-sponsored certifications | Conference attendance | Continuing education | Flexible working hours | Paid time offSenior-level Full TimeSan Luis Potosi, Mexico R1d ago
-
Access Control | Automated scaling | Azure Data | Azure Data Factory | Azure Data LakeCompany-sponsored certifications | Conferences | Continuing education | Flexible working hours | Paid time offSenior-level Full TimeSaltillo, Coahuila, Mexico R1d ago
-
Access Control | Agile | Azure Data | Azure Data Factory | Azure Data LakeCompany Sponsored Coursework Exams and Certifications | Continuing education | Flexible working hours | Paid time off | Remote workSenior-level Full TimePuebla City, Mexico R1d ago
-
Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure DevOpsCompany-sponsored certifications | Conference attendance | Continuing education | Flexible working hours | Paid time offSenior-level Full TimeNaucalpan, Mexico R1d ago
-
Access Control | Agile | Azure Data | Azure Data Factory | Azure Data LakeCompany-sponsored certifications | Company-sponsored coursework | Continuing education | Flexible working hours | Paid time offSenior-level Full TimeMérida, Yucatán, Mexico R1d ago
-
Azure | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake StorageCompany-sponsored certifications | Conferences | Continuing education | Flexible working hours | Paid time offSenior-level Full TimeMonterrey, Nuevo León, Mexico R1d ago
-
Azure | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake StorageCompany-sponsored certifications | Continuing education | Flexible working hours | Paid time off | Remote workSenior-level Full TimeMexico City, Mexico R1d ago
-
Agile | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake StorageCompany Sponsored Coursework Exams and Certifications | Continuing education training conferences | Flexible working hours | Paid time off | Remote workSenior-level Full TimeMexicali, Baja California, Mexico R1d ago
-
Agile | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake StorageCompany-paid certifications | Conferences | Continuing education | Flexible working hours | Paid time offSenior-level Full TimeLeón, Guanajuato, Mexico R1d ago
-
Access Control | Automation | Azure Data | Azure Data Factory | Azure Data LakeCompany-sponsored certifications | Conference opportunities | Continuing education | Flexible working hours | Paid time offSenior-level Full TimeHermosillo, Sonora, Mexico R1d ago
-
Agile | Azure | Azure Data | Azure Data Factory | Azure Data LakeCompany-sponsored certifications | Conference attendance | Continuing education | Flexible working hours | Paid time offSenior-level Full TimeGuadalajara, Jalisco, Mexico R1d ago