Tech Lead - Data Engineering

Tech Lead - Data Engineering-079065

Description

 

Job Title

Tech Lead - Data Engineering

Location 

Vancouver

Position Overview:
We are seeking a highly skilled, experienced and hands-on Senior Data Engineer to serve as a Tech Lead in a critical data implementation project. The ideal candidate will have 7 to 10 years of experience in ETL, Databricks, Python, Spark, and SQL databases and a strong understanding of data ingestion, transformation, and implementation in complex environments. This role requires expertise in DevOps and CI/CD for data, advanced SQL and Python programming skills, and hands-on experience with modern data engineering tools and architectures.

Responsibilities:

• Serve as a Tech Lead in a data implementation project, guiding architecture decisions and best practices.
• Design, build, and optimize ETL/ELT pipelines for batch and streaming data using Databricks, PySpark, Python, and SQL.
• Implement dimensional modeling and database normalization techniques for efficient data storage and retrieval.
• Lead database development, including DDL and DML scripting, query optimization, and performance profiling across platforms like Spark, PostgreSQL, and Databricks SQL.
• Support the Development and maintaining of CI/CD pipelines for data workflows using Terraform, Kubernetes, Argo, or Cross Plane.
• Work with DBT to manage models, seeds, environments, macros, unit tests, and incremental materialization.
• Implement data versioning and migration strategies using Alembic (preferred), Flyway, or Liquibase.
• Optimize data ingestion and transformation for structured and unstructured data sources.
• Design and implement real-time streaming solutions using Azure EventHub or similar.
• Manage infrastructure as code and containerized environments with Docker and Kubernetes.
• Monitor, troubleshoot, and optimize workflows using logging and querying tools like KQL (Azure) or similar.
• Collaborate with data scientists, analysts, and business teams to ensure data solutions align with business needs.

Skills & Qualifications:

• Expertise in Python: Strong proficiency, unit testing experience, and deep knowledge of libraries such as Pandas, NumPy, SQLAlchemy, and Alembic.
• Advanced SQL skills: Proficiency in CTEs, subqueries, joins, window functions (e.g., LAG), indexing strategies, and performance tuning.
• Databricks Experience: Proficiency with Workflows/Jobs, Clusters, SQL Warehouse, Unity Catalog, performance profiling, and log analysis.
• Spark Expertise: Hands-on experience with PySpark, Spark SQL, batch and streaming processing, delta tables, and parquet.
• Database Proficiency: Strong understanding of PostgreSQL (queries, indexes, JSON columns), SQL and NoSQL database selection.
• Dimensional Modeling & Data Architecture: Deep knowledge of fact/dimension tables and normalization (Normal Form).
• Streaming Data Processing: Experience with Azure EventHub or equivalent streaming technologies.
• DevOps & CI/CD for Data: Experience with Kubernetes, Terraform, Argo, and Cross Plane for infrastructure as code.
• Data Testing & Quality: Familiarity with unit tests, data tests, and snapshots in DBT.
• Logging & Monitoring: Ability to query logs using KQL (Azure) or similar tools.

Life at Capgemini

Capgemini supports all aspects of your well-being throughout the changing stages of your life and career. For eligible employees, we offer:

• Flexible work
• Healthcare including dental, vision, mental health, and well-being programs
• Financial well-being programs such as 401(k) and Employee Share Ownership Plan
• Paid time off and paid holidays
• Paid parental leave
• Family building benefits like adoption assistance, surrogacy, and cryopreservation
• Social well-being benefits like subsidized back-up child/elder care and tutoring
• Mentoring, coaching and learning programs
• Employee Resource Groups
• Disaster Relief

About Capgemini

Capgemini is a global business and technology transformation partner, helping organizations to accelerate their dual transition to a digital and sustainable world, while creating tangible impact for enterprises and society. It is a responsible and diverse group of 340,000 team members in more than 50 countries. With its strong over 55-year heritage, Capgemini is trusted by its clients to unlock the value of technology to address the entire breadth of their business needs. It delivers end-to-end services and solutions leveraging strengths from strategy and design to
engineering, all fueled by its market leading capabilities in AI, generative AI, cloud and data, combined with its deep industry expertise and partner ecosystem. The Group reported 2024 global revenues of €22.1 billion.

Get the future you want | www.capgemini.com

Disclaimer

Capgemini is an Equal Opportunity Employer encouraging diversity in the workplace. All qualified applicants will receive consideration for employment without regard to race, national origin, gender identity/expression, age, religion, disability, sexual orientation, genetics, veteran status, marital status or any other characteristic protected by law.

This is a general description of the Duties, Responsibilities and Qualifications required for this position. Physical, mental, sensory or environmental demands may be referenced in an attempt to communicate the manner in which this position traditionally is performed. Whenever necessary to provide individuals with disabilities an equal employment opportunity, Capgemini will consider reasonable accommodations that might involve varying job requirements and/or changing the way this job is performed, provided that such accommodations do not pose an undue hardship.

Capgemini is committed to providing reasonable accommodations during our recruitment process. If you need assistance or accommodation, please reach out to your recruiting contact.

Please be aware that Capgemini may capture your image (video or screenshot) during the interview process and that image may be used for verification, including during the hiring and onboarding process.

Click the following link for more information on your rights as an Applicant http://www.capgemini.com/resources/equal-employment-opportunity-is-the-law

Applicants for employment in the US must have valid work authorization that does not now and/or will not in the future require sponsorship of a visa for employment authorization in the US by Capgemini.

    

Job

 : Programmer/Analyst

Schedule

 : Full-time

Primary Location

 : CA-BC-Vancouver

Organization

 : I&D