AI Test Engineer - Senior Manager
Tasks
- Build and maintain ground truth datasets
- Build evaluation frameworks
- Define classification and extraction quality standards
- Define standardized AI evaluation patterns
- Design adversarial and boundary test inputs
- Design drift detection frameworks
- Develop parameterized evaluation scripts
- Embed evaluations into CI CD pipelines
- Enable auditability traceability and transparency
- Ensure responsible AI data privacy governance compliance
- Establish regression thresholds for release gating
- Identify risks and performance gaps and escalate
- Implement AI as judge scoring
- Perform API testing of AI endpoints
- Test end to end agent workflows
- Translate AI testing strategy into test scenarios
- Translate evaluation results into actionable insights
- Update datasets for evolving requirements
- Validate AI outputs against performance thresholds
- Validate data persistence using SQL
Perks/Benefits
- N/A
Skills/Tech-stack
AI Foundry | API Testing | AWS Bedrock | Adversarial Testing | Azure AI | Azure AI Foundry | CI/CD | Data Validation | Document Classification | Drift Detection | HTTPx | Hallucination detection | LLM Evaluation | Langchain | Language Models | Language Processing | Large Language Models | Llamaindex | Machine Learning | Natural Language | Natural Language Processing | Newman | OCR | Postman | Prompt engineering | Python | RAG | Requests | Retrieval-Augmented Generation | SQL | Statistical Analysis
Education
Bachelor of Engineering | Bachelor of Science | Bachelor of Science in Computer Science | Bachelor of Science in Data Science
Roles
AI | AI Test Engineer | Engineer | Senior AI Test Engineer | Test Engineer
Related jobs
-
Staff Software Engineer, AI Garage INR 2000K-4800KAgentic AI | Artificial Intelligence | C# | C++ | Distributed SystemsSenior-level Full TimeHyderabad, Telangana, India4h ago
-
Senior Staff Engineer (AI Developer - AppSec) INR 2000K-4500KAPI Security | API Security Top 10 | API security testing | Agentic Workflows | Azure DevOpsSenior-level Full TimeMumbai, India5h ago
-
Senior Staff Engineer (AI Developer - DevSecOps Tools) INR 3000K-5000KAKS | ARM Templates | AWS EventBridge | Aqua Security | ArgoCDSenior-level Full TimeMumbai, India6h ago
-
Senior Data Engineer INR 2520K-3380KApache Airflow | Apache Iceberg | Azure Data | Azure Data Lake | Azure DevOpsSenior-level Full TimePune, MH, IN15h ago
-
Software Engineer II-Continous Learning INR 2000K-3500KAsynchronous programming | Containerization | Distributed Systems | Docker | Event StreamingConference reimbursement | Employee assistance program | Employee stock purchase program | Flexible time off | LinkedIn LearningMid-level Full TimeHyderabad16h ago
-
Software Engineer I-Continuous Learning INR 700K-800KAsynchronous programming | C++ | Containerization | Distributed Systems | DockerConference reimbursement | Employee assistance program | Employee stock purchase program | Equity compensation | Flexible time offEntry-level Full TimeHyderabad16h ago
-
Staff Software Engineer-AI INR 2000K-4500KAsynchronous Architecture | Autogen | Backpressure | CrewAI | Data GenerationEmployee assistance program | Flexible time off | LinkedIn Learning subscription | Local Employee Meetups | Training and education reimbursementSenior-level Full TimeHyderabad16h ago
-
ETL Test Engineer (3-6 yrs) (Pune/Chennai) - (Must Have- ETL Testing, Python, Pyspark, Azure Databricks) INR 900K-1500KAPI Testing | Apache Airflow | Azure Databricks | Bitbucket | DockerEmployee Assistance Program (EAP) | Flexible working environment | LinkedIn Learning | Volunteer time offMid-level Full TimeChennai/Pune, India22h ago
-
AI Researcher INR 2000K-3500KAlgorithmic Fairness | Bias Audit | Data Annotation | Dataset Building | Educational measurementMid-level Full TimeHybrid in Bangalore, India R1d ago
-
Data Engineer - HR Analytics & Workday Prism INR 1500K-2000KAPI Integrations | AWS SageMaker | Agile | Apache Spark | AzureMid-level Full TimePune, Maharashtra, India1d ago
-
Data Bricks Engineer INR 1500K-2400KDatabricks | Databricks SQL | Delta Lake | JSON-RPC | MLflowWork from office 5 days per weekSenior-level Full TimeGurugram, Haryana, India1d ago
-
Staff Engineer ( Meshing, C++ / C#, GenAI ) INR 1753K-2800KC# | C++ | CUDA | Computational Geometry | DebuggingHealth insurance | Wellness benefitsSenior-level Full TimeIndia Off-Site, India1d ago
-
Mid-level Full TimeChennai, Tamil Nadu, India1d ago
-
AI/ML Silicon Validation Lead, Google Cloud, TPU INR 3200K-5000KAXI | Board Schematic | Board Schematic Analysis | Bring-up | DDRSenior-level Full TimeBengaluru, Karnataka, India1d ago
-
Engineer - Embedded Automation QA INR 480K-1050KAPI Testing | Azure DevOps | BACNet IP | C plus plus | CI/CDMid-level Full TimeHadapsar, Pune, Maharashtra, IND, 4110131d ago
-
Data Analytics Professional INR 1500K-2700KAPIs | Agile | Cloud Data | Cloud Data Platforms | ConfluenceMid-level Full TimeBengaluru, IN, 5601031d ago
-
GCP Data Engineer/Consultant Specialist INR 1500K-2500KAgile | Airflow | Apache Kafka | Apache NiFi | BigQueryFlexible working | Opportunities for growth | Professional developmentSenior-level Full TimeHyderabad, Telangana, India R1d ago
-
Senior MLOps Engineer INR 2520K-3500KAWS Lambda | AWS Step Functions | Airflow | Amazon SageMaker | CI/CDDental insurance | Employee assistance program | Flex Time Off | Health insurance | Hybrid work scheduleSenior-level Full TimeHyderabad, Telangana, India1d ago
-
Senior AI Engineer INR 3000K-5000KAPI deployment | Adversarial Testing | Auto Retraining | Azure ML | CI/CDSenior-level Full TimeBangalore1d ago
-
Azure Data | Azure Data Factory | CI/CD | Data Factory | Data ReconciliationSenior-level Full TimeBangalore, India1d ago
-
Amazon Web Services | Batch ETL | Big Data | Data Architecture | Data GovernanceSenior-level Full TimeMumbai, Mahārāshtra, India1d ago
-
Senior Talend Developer / Lead Data Engineer INR 1500K-2700KAzure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure DevOpsSenior-level Full Timebengaluru, India1d ago
-
AWS Data Engineer (Senior) (Freelancer) INR 2040K-3000KAmazon Athena | Amazon EMR | Amazon Glue | Amazon Redshift | Apache AirflowSenior-level FreelanceMumbai, MH R1d ago
-
Solutions Architect INR 2400K-3500KADLS | AWS | Airflow | Azure | CassandraBootcamp training | Broadband reimbursement | Continuous learning opportunities | Medical insurance | Paid certificationsSenior-level Full TimeIndia - Remote R1d ago
-
Sr. SW Engineer, Machine Learning INR 2475K-3465KA/B | A/B Testing | Agent systems | B testing | CI/CDFinancial wellness support | Hybrid work | Mental health support | Time offSenior-level Full TimeBengaluru, India1d ago