AI Engineer – AI Observability and Quality
Tasks
- Build automated test suites
- Create evaluation frameworks
- Define evaluation standards and practices
- Design evaluation infrastructure
- Develop regression detection systems
- Measure response accuracy
- Monitor cost and latency
- Operate independently and set roadmap
- Translate quality criteria into test scenarios
- Validate citation correctness
Perks/Benefits
- Company paid cell phone plan
- Dental insurance
- Employee assistance program
- Flexible work hours
- Health insurance
- Life insurance
- Long-term disability insurance
- New Parenting Leave
- Paid sick leave
- Paid vacation days
- Part-time option
- Remote work option
- Retirement savings plan with match
- Second medical opinion
- Technology allowance
- Vision insurance
Skills/Tech-stack
A/B | A/B Testing | B testing | DeepEval | Evaluation Frameworks | Generative AI | Information Retrieval | LLM Evaluation | Langfuse | Langsmith | Language Models | Large Language Models | Machine Learning | Python | RAG | Ragas | Significance Testing | Statistical significance | Statistical significance testing | Weights and Biases
Education
N/A
Related jobs
-
Computer Vision, AI (Co-op) CAD 51K-64KComputer Vision | Data Analysis | Data cleaning | Data labeling | Edge ComputingEntry-level Full TimeSaskatoon, SK, CA, S7K 5R614h ago
-
Computer Vision | Document processing | Edge AI | Intelligent agents | Language ModelsEntry-level Full TimeSaskatoon, SK, CA, S7K 5R614h ago
-
Senior Software Engineer, Data CAD 247K-375KAWS DMS | AWS Glue | AWS RDS | Amazon Kinesis | Amazon RedshiftFlexible work hours | Health/dental coverage | Parental leave top-ups | Remote work | Unlimited PTOSenior-level Full TimeCanada (Remote) R15h ago
-
Advisor, Data engineering CAD 60K-90KCloud Data | Cloud Data Platforms | Data Engineering | Data Mining | Data ModelingFlexible vacation | Group insurance | Health and wellness reimbursement | Pension plan | TelemedicineMid-level Full TimeComplexe Desjardins Montréal, Canada20h ago
-
Product Reliability Engineer (Analytics) CAD 96K-127KData Analysis | KPI Reporting | Microsoft 365 | Python | SQLEmployee assistance program | Hybrid work | Learning and career development | Paid time off | Transit subsidyMid-level Full TimeVancouver20h ago
-
Staff Analytics Engineer CAD 120K-147KAWS Step Functions | Access Control | Airbyte | Apache Airflow | Cost OptimizationPaid time off | Professional growth opportunities | Supportive, inclusive work environmentSenior-level Full TimeCanada1d ago
-
Analytics Engineer CAD 115K-135KAutomated testing | BigQuery | CI/CD | Containerization | DBTComputer purchase program | Employee technology purchase program | Employer pension contribution | Enhanced parental leave | Equity optionsMid-level Full TimeMontreal, Quebec, Canada R1d ago
-
Senior Data Engineer, Analytics Platform CAD 90K-110KAzure Data | Azure Data Factory | CI/CD | Data Factory | Data Governance4 weeks vacation | Best-in-class tools | Paid health insurance | Teamwork Friendly Working Framework | Unlimited mental health servicesSenior-level Full TimeVancouver, Canada1d ago
-
Senior Data Engineer, Analytics Platform CAD 90K-110KAzure Data | Azure Data Factory | CI/CD | Data Factory | Data GovernanceBest-in-class tools | Career growth | Collaborative work environment | Paid vacation | Premium insuranceSenior-level Full TimeVancouver, Canada1d ago
-
Intermediate AI/ML Engineer CAD 98K-125KAWS | Computer Vision | Data Pipelines | Experimentation | JavaScriptHealth spending account | Health/dental coverage | Hybrid work | Paid time off | Wellness reimbursementMid-level Full TimeOttawa Office1d ago
-
Senior-level Full TimeToronto, Canada1d ago
-
Spécialiste en intégration robotique CAD 104K-130KABB | Control Systems | Fanuc | Industrial Automation | Machine ControlCareer development plan | Community engagement events | Comprehensive health insurance | Hybrid work | Mentorship programSenior-level Full TimeMontréal, QC, Canada1d ago
-
Prompt Engineer CAD 110K-140KAgile | Data Visualization | Hugging Face | Language Models | Language ProcessingMid-level Full TimeToronto , Canada1d ago
-
AWS Technical Lead - Data & Analytics CAD 140K-160KAWS | AWS CloudFormation | AWS Glue | AWS Lambda | AWS Step FunctionsCareer advancement | Comprehensive benefits plan | Employee referral program | Flexible work environment | Hybrid work scheduleSenior-level Full TimeMarkham, ON, Canada1d ago
-
Bash | Cloud platform | Data Processing | Docker | Google CloudAsynchronous culture | Remote-friendly environment | Supportive teamMid-level Full TimeOttawa, Canada1d ago
-
Bash | Data Ingestion | Data Processing | Docker | GCPAsynchronous culture | Career growth opportunities | Friendly work environmentMid-level Full TimeCalgary, Canada1d ago
-
Senior AI Engineer CAD 129K-179KAI Foundry | AI Search | Agentic AI | Azure | Azure AIDental insurance | Employee assistance program | Health insurance | Hybrid work | Life insuranceSenior-level Full TimeCAN, Ontario, Toronto, 200 Bloor Street …1d ago
-
Data Engineer CAD 105K-129KApache Spark | Azure Data | Azure Data Factory | CI/CD | Cost OptimizationSenior-level Full TimeTD Centre - TD Tower - …1d ago
-
Data Engineer CAD 90K-100KAzure Data | Azure Data Factory | Azure Data Lake | Cloud Architecture | Data FactoryCoaching opportunities | Dental insurance | Flexible hybrid policy | Health insurance | Higher education certification policySenior-level Full TimeMississauga, ON, Canada1d ago
-
Senior Data Engineer CAD 126K-164KApache Spark | Azure Data | Azure Data Factory | CI/CD | Cost OptimizationSenior-level Full TimeTD Centre - TD Tower - …1d ago
-
Data Engineer with OpenShift CAD 100K-115KData Governance | Data Modeling | Data Privacy | Data Security | Data WarehousingFlexible hybrid policy | Higher education certification policy | Insurance (medical | Insurance dental | Insurance visionMid-level Full TimeMississauga, ON, Canada1d ago
-
Senior Backend Developer, Analytics Pipeline CAD 82K-106KAdobe Analytics | Amplitude | Apache Beam | Apache Dataflow | Apache HadoopDefined benefits pension plan | Employee assistance for diversity and inclusion | Employee resource groups | Flexible work schedule | Health benefitsSenior-level Full TimeToronto (36.25), Canada1d ago
-
Big Data Engineer CAD 85K-95KAWS | AWS Glue | AWS Lake Formation | AWS S3 | AgileDental coverage | Education assistance | Group retirement savings plan | Gym access | Health insuranceEntry-level Full TimeLOC0197 Toronto Downtown Branch, Canada1d ago
-
AWS Batch | AWS EC2 | AWS S3 | Agile | ArtifactoryRelocation assistanceSenior-level Full TimeLaval, Quebec, Canada1d ago
-
Lead Software Engineer (Python | Databricks) CAD 94K-171KAgile | Apache Airflow | Apache Kafka | Apache Spark | AzureAccessibility accommodations | Community engagement programs | Cross-functional development opportunities | Employee resource groups | Flexible vacationSenior-level Full TimeToronto, ON, CA, M5H1H11d ago