Lead Data Engineer

Job Summary:

We are seeking an experienced and dynamic Technical Lead for Data Engineering to join our team. This role involves leading the architecture, design, development and management of our data infrastructure, collaborating across teams, and driving best practices in data engineering. You will play a critical role in shaping and delivering scalable, high-performance data solutions for our business.

Key Responsibilities:

Technical Leadership:

• Provide technical guidance and leadership to a team of data engineers.
• Define and maintain best practices, standards, and architecture for data engineering.
• Drive innovation, quality and continuous improvement in data pipelines and infrastructure.

Data Infrastructure Development:

• Design, build, and maintain scalable and robust data pipelines for batch and real-time processing.
• Establish and maintain robust data lineage tools to track the flow of data from the source to the destination.
• Optimize data storage and retrieval strategies for performance and cost-effectiveness.
• Ensure data quality, consistency, and reliability through automated testing and monitoring.
• Provide the tooling for easy data access. 
• Establish process and tool for data versioning.

Collaboration & Strategy:

• Collaborate with data scientists, analysts, and business stakeholders to understand data needs and provide easy access to data to create actionable insights.
• Partner with product and engineering teams to integrate data solutions into the core logistics systems.
• Translate business objectives into technical solutions by designing and implementing strategic data initiatives.
• Design a self-service data platform where teams can easily access the data they need without compromising security or overwhelming the data infrastructure.

Project Management:

• Lead project planning, estimation, and execution to ensure timely delivery.
• Manage resources and allocate work across the team to balance priorities and deadlines.
• Use agile frameworks to plan and prioritize the execution of data projects.
• Communicate project updates and technical concepts to both technical and non-technical audiences.

Requirements

• Practical experience in Computer Science, Engineering, or a related field.
• 5+ years of experience in data engineering, with at least 2 years in a leadership or senior role.
• Strong proficiency in programming languages, preferably in Python.
• Expertise in SQL and working with relational and non-relational databases (e.g., PostgreSQL, MongoDB, Cassandra).
• Hands-on experience with big data technologies (e.g. BigQuery, Hadoop, Spark) and cloud platforms, preferably GCP.
• Deep understanding of ETL/ELT processes, data modeling, and data warehousing concepts.
• Experience with orchestration tools.
• Knowledge of DevOps practices, CI/CD pipelines, and infrastructure-as-code tools.
• Strong analytical and problem-solving skills with attention to detail.
• Excellent communication and interpersonal skills.

Preferred Qualifications:

• Experience with streaming data technologies.
• Knowledge of machine learning pipelines and frameworks.
• Certifications in cloud platforms.
• Familiarity with data governance and compliance frameworks (e.g., GDPR, CCPA).