Lead Spark Data Engineer
Tasks
- Architect data solutions on Spark and Flink
- Build Query Validator
- Develop Spark Adapter
- Enforce data governance, security, and quality practices
- Ensure logic parity between Spark and Flink
- Implement IoT Query Language grammar
- Implement relationships logic within core
- Maintain documentation
- Manage Azure and Databricks resources
- Monitor and optimize workloads
- Participate in agile activities
- Transform, clean, and prepare data with SQL, Python, Java
Perks/Benefits
- N/A
Skills/Tech-stack
ANTLR | Abstract syntax trees | Apache Spark | Azure DevOps | CI/CD | Data Governance | Data Modeling | Data Security | Data Warehousing | Databricks | Delta Lake | Flink | GitHub | Java | Python | SQL
Education
N/A
Roles
Related jobs
-
API Integration | Agent Frameworks | Artificial Intelligence | Backend Services | Content ModerationCollaborative team environment | High impact AI applications | Modern development tools and resources | Remote work flexibilitySenior-level Full TimeCanada R20h ago
-
AWS | Agile | Azure | Cloud Computing | Data Governance100 percent remote | Career growth | Continuous learning | Innovation culture | International projectsSenior-level Full TimeCanada R20h ago
-
ML Engineer, II - Road & Lane USD 139K-183KBEV | CUDA | CUDA kernels | Camera Calibration | Computer VisionMid-level Full TimeRemote - US, Ann Arbor, MI, … R1d ago
-
ML Engineer, II - Learned Behaviors USD 153K-222KBehavior Cloning | Data Pipelines | Distributed Training | Graph Neural Networks | Imitation LearningMid-level Full TimeRemote - US, Ann Arbor, MI, … R1d ago
-
ML Engineer, II - End to End (E2E) USD 153K-183KBEV | Computer Vision | Data workflows | Diffusion Models | Distributed TrainingMid-level Full TimeRemote - US, Ann Arbor, MI, … R1d ago
-
ML Engineer, II - Camera Models USD 153K-202K3D Perception | Camera Calibration | Computer Vision | Convolutional Neural Networks | Deep learningMid-level Full TimeRemote - US, Ann Arbor, MI, … R1d ago
-
ML Engineer, II - Birds Eye View (BEV) USD 150K-190KBEV Representations | Birds Eye View | Cameras | Computer Vision | Data PipelinesMid-level Full TimeRemote - US, Ann Arbor, MI, … R1d ago
-
BEV | Cameras | Computer Vision | Data Analysis | Data DistributionCommute subsidy | Flexible schedule | Life insurance | Medical, dental & vision coverage | Office Closures on HolidaysEntry-level ApprenticeshipRemote - US, Ann Arbor, MI, … R1d ago
-
Senior Solutions Engineer (Canada) CAD 120K-140KAPI Integration | Apache Spark | Cloud Operations | Data Pipelines | Data ValidationDental insurance | Flexible hours | Flexible-hybrid work | Health insurance | Parental leaveSenior-level Full TimeRemote - Calgary, Alberta, Canada R2d ago
-
Sr. AI Engineer CAD 180K-230KAPI Integration | Backend Services | Content Moderation | Entity Resolution | EvaluationSenior-level Full TimeRemote - Canada R2d ago
-
Senior Data Engineer CAD 150K-175KCCPA | Data Lakes | Data Modeling | Data Quality | Data ValidationAnnual company retreats | Home office support | Remote-firstSenior-level Full TimeToronto - Remote R3d ago
-
Bash | C# | C++ | Coverage.py | DockerFlexible hours | Project-based collaboration | Remote workSenior-level Full TimeCanada - Remote R3d ago
-
Data Manipulation | Data Preprocessing | Data Visualization | GenAI | LLMsCareer development | Flexible schedule | Freelance | Part-time | Remote workMid-level FreelanceCanada - Remote R3d ago
-
Machine Learning Engineer CAD 123K-192KDeep learning | Gradient descent | Graph theory | Language Models | Large Language ModelsMid-level Full TimeToronto - MSO, Canada R3d ago
-
Développeur DataOps Senior CAD 150K-250KAWS RDS | Data replication | Linux | Observation tools | Partitioning401k plan | Flexible paid time off | Learning and development allowance | Medical/Dental/Vision insurance | Paid parental leaveSenior-level Full TimeCanada (Remote) R3d ago
-
Applied AI Engineer, Internal Products CAD 70K-113KAI Agents | API Integrations | Anthropic API | Automation | Code ToolsRemote-first flexibilityEntry-level Full TimeRemote (Canada), Remote (US) R3d ago
-
Sr. AI Engineer (24 months fixed-term) USD 137K-232KBias Mitigation | Communication | Data Analysis | Education Technology | Generative AIFlexible schedule | Health insurance | Inclusive culture | Learning and development opportunities | Paid time offSenior-level TemporaryMountain View, CA / Remote (Continental … R4d ago
-
Senior Data Engineer CAD 133K-166KAWS | Cloud Architecture | Data Governance | Data Modeling | Data PipelinesSenior-level Full TimeCanada Remote - Consultant use only R4d ago
-
Bash | C# | Coverage | Docker | GoFlexible participation | Fully remote | Project-based collaborationSenior-level Full TimeCanada - Remote R4d ago
-
Freelance Machine Learning Engineer CAD 110KComputational Algorithms | Mathematical Modeling | NumPy | Pandas | Problem SolvingFlexible schedule | Professional development | Remote workMid-level FreelanceCanada - Remote R4d ago
-
Data Engineer - Snowflake CAD 130K-166KAWS Cloud | Apache Spark | Data Lake | Data Modeling | Data TransformationCareer development opportunities | Small entrepreneurial environmentSenior-level Full TimeToronto, Ontario, Canada - Remote R4d ago
-
Senior Data Platform Engineer CAD 129K-152KAPI Gateway | AWS | Batch Loading | CI/CD | Change Data CaptureHealth insurance | Parental leave | Professional development stipend | Remote workSenior-level Full TimeRemote - Canada R4d ago
-
Senior AI Engineer USD 180K-298KA/B | A/B Testing | API Integration | AWS | Agent systems401k plan | Employee assistance program | Flex PTO | Global travel coverage | Medical/Dental/Vision insuranceSenior-level Full TimeRemote, Canada; Remote, United States R4d ago
-
Senior DevOps Engineer | MStar CAD 111K-160KAWS | Build systems | C++ | CI/CD | Cloud infrastructureCollaborative team | Innovative projects | Remote-friendly environmentSenior-level Full TimeRemote - Canada R5d ago
-
Senior / Staff Perception Engineer USD 158K-269K3D Detection | Computer Vision | Deep learning | Machine Learning | Model DeploymentCatered meals | Competitive compensation | Equity awards | Flexible hours | Health insuranceSenior-level Full TimeRemote US & Canada R5d ago