Staff Data Engineer
Toronto, Ontario
ShyftLabs
Transform your business with Shyftlabs' cutting-edge AI solutions, digital transformation services, and expert technology consulting. Drive innovation and growth with our proven expertise.
Position Overview:ShyftLabs is seeking an experienced Staff Data Engineer to implement and optimize enterprise data pipelines and ingestion frameworks for Fortune 500 companies. You will build robust Databricks-based data processing infrastructure, implement pipeline reliability solutions, and create scalable technical solutions for both batch and streaming data workloads.
ShyftLabs is a growing data product company founded in early 2020 and works primarily with Fortune 500 companies. We deliver digital solutions built to help accelerate the growth of businesses in various industries, by focusing on creating value through innovation.
ShyftLabs is an equal-opportunity employer committed to creating a safe, diverse and inclusive environment. We encourage qualified applicants of all backgrounds including ethnicity, religion, disability status, gender identity, sexual orientation, family status, age, nationality, and education levels to apply. If you are contacted for an interview and require accommodation during the interviewing process, please let us know.
ShyftLabs is a growing data product company founded in early 2020 and works primarily with Fortune 500 companies. We deliver digital solutions built to help accelerate the growth of businesses in various industries, by focusing on creating value through innovation.
Job Responsibilities:
- Implement and optimize enterprise-scale data pipeline architecture processing 1500+ daily jobs across multiple business domains
- Build and maintain Databricks job orchestration solutions integrated with Active Batch scheduling platforms
- Implement data ingestion frameworks supporting CDC via AWS DMS and S3-based data lake patterns
- Troubleshoot and resolve pipeline reliability issues including overrun problems, dependency failures, and compute optimization
- Build observability and monitoring solutions for large-scale data operations teams
- Implement near real-time ingestion patterns using DLT (Delta Live Tables) and streaming architectures
- Build and maintain data quality frameworks integrated with data governance processes
- Implement serverless migration strategies and optimize cloud resource utilization
- Lead technical workshops and provide hands-on guidance to client engineering teams
- Enterprise Pipeline Expertise: Proven experience building and optimizing data pipelines at Fortune 500 scale
- Operational Excellence: Deep understanding of pipeline monitoring, alerting, and failure recovery patterns
- Performance Optimization: Ability to analyze and optimize compute resources and job execution patterns
- Technical Consulting: Strong workshop facilitation and client relationship skills, with ability to present technical solutions and lead hands-on training sessions with client engineering teams
Basic Qualifications:
- Bachelor's or Master's degree in Computer Science, Engineering, or related technical field
- 6+ years of experience in enterprise data engineering with Fortune 500 companies
- Expert-level experience with Databricks platform including job scheduling and cluster management
- Hands-on experience with Active Batch or similar enterprise job orchestration platforms
- Deep knowledge of AWS data services including DMS, S3, and database connectivity
- Proven experience with CDC (Change Data Capture) implementations at enterprise scale
- Strong background in Delta Live Tables (DLT) and streaming data processing
- Understanding of Apache Spark optimization and performance tuning
- Experience with enterprise database integration and cross-platform data connectivity
Preferred Qualifications
- Experience with enterprise integration patterns and API data ingestion
- Knowledge of cloud storage optimization and data lake best practices
- Background in high-volume transaction processing systems
- Experience with retail or energy industry data patterns
- Familiarity with trading and pricing systems data workflows
ShyftLabs is an equal-opportunity employer committed to creating a safe, diverse and inclusive environment. We encourage qualified applicants of all backgrounds including ethnicity, religion, disability status, gender identity, sexual orientation, family status, age, nationality, and education levels to apply. If you are contacted for an interview and require accommodation during the interviewing process, please let us know.
* Salary range is an estimate based on our AI, ML, Data Science Salary Index 💰
Job stats:
0
0
0
Categories:
Engineering Jobs
Leadership Jobs
Tags: APIs Architecture AWS Computer Science Consulting Databricks Data governance DataOps Data pipelines Data quality Engineering Pipelines Spark Streaming
Perks/benefits: Career development Competitive pay Insurance Startup environment
Region:
North America
Country:
Canada
More jobs like this
Explore more career opportunities
Find even more open roles below ordered by popularity of job title or skills/products/technologies used.
Principal Data Engineer jobsPower BI Developer jobsSr. Data Engineer jobsBI Developer jobsStaff Data Scientist jobsStaff Machine Learning Engineer jobsPrincipal Software Engineer jobsData Science Intern jobsDevOps Engineer jobsJunior Data Analyst jobsData Science Manager jobsSoftware Engineer II jobsData Manager jobsData Analyst Intern jobsLead Data Analyst jobsStaff Software Engineer jobsBusiness Data Analyst jobsAccount Executive jobsData Specialist jobsAI/ML Engineer jobsSr. Data Scientist jobsData Governance Analyst jobsBusiness Intelligence Analyst jobsSenior Backend Engineer jobsData Engineer III jobs
Consulting jobsMLOps jobsAirflow jobsOpen Source jobsLinux jobsEconomics jobsKPIs jobsJavaScript jobsKafka jobsTerraform jobsGitHub jobsData Warehousing jobsPostgreSQL jobsRDBMS jobsScikit-learn jobsNoSQL jobsComputer Vision jobsStreaming jobsClassification jobsBanking jobsPhysics jobsGoogle Cloud jobsPrompt engineering jobsRAG jobsOracle jobs
Pandas jobsHadoop jobsR&D jobsBigQuery jobsData warehouse jobsScala jobsdbt jobsLooker jobsGPT jobsReact jobsScrum jobsPySpark jobsDistributed Systems jobsLangChain jobsCX jobsELT jobsIndustrial jobsMicroservices jobsSAS jobsJira jobsRedshift jobsOpenAI jobsJenkins jobsModel training jobsRobotics jobs