Lead Data Engineer

Octopus Energy supplies electricity and gas to over 7 million UK householders and is at the forefront of innovation in the drive towards net zero. Our Gross Margin team produces key financial reporting and analysis to support decision making and margin improvements across the UK business. We also support teams in other countries where Octopus operates.
The Gross Margin pipeline - for which you will be responsible along with a team of three engineers - uses a vast amount of data and produces a highly granular view of consumption, revenue and costs. Challenges include the quantity and quality of data and the estimations required where data does not exist. We have a superb pipeline with a very exciting dataset, analysis from which provides valuable information s across the business, e.g. driving development of Octopus’ industry-leading Time Of Use tariffs, refining financial forecasting, and preventing multi-million pound write-offs. 
The Lead Data Engineer will drive the technical vision behind this pipeline as our customer base continues to grow and market development towards net zero requires an increasingly data-heavy calculation of energy consumption, and will also take joint responsibility for the accuracy of our outputs and guiding engineers in the team on their own career progression.

What You'll Do...

• Product Leadership: You will shape the design of the Gross Margin pipeline so it continues to accurately reflect actual revenue and costs when reconciled against internal billing systems and external industry flow data, whilst also making it more accessible and valuable to teams across the wider Octopus business.
• People Leadership: You will support and develop the technical progression of top-tier engineering talent while fostering an inclusive and productive team environment.
• Technical Leadership: You will oversee the technical and delivery outcomes of our Gross Margin pipeline for maximum scalability, efficiency and simplicity whilst driving best practices within the codebase. 

What You'll Need...

• Proven Track Record: History of shipping product updates, getting things done, and leading teams in high-growth, product-focused organisations.
• Management and Leadership Experience: 2+ years managing high-performance engineering teams, with a focus on people leadership, providing support, coaching, and development.
• Modern Data Expertise: 5+ years building highly scalable technology products using SQL and Python. Experience with Databricks, dbt, Spark and AWS would be a big plus.
• Commercial Mindset: Experience in financial reporting is not required (although a plus), but appreciation for the business implications of what the data shows us is key.
• Test Driven Development: Passion for ensuring code changes improve our view of Gross Margin and don’t inadvertently break anything else.
• Hands-on Approach: You are eager to contribute to design and code alongside your team.
• Engineering Excellence: Fostering a high bar for engineering excellence, holding teams accountable for technical outcomes and individual growth.
• Balanced Decision-Making: Ability to make trade-offs between product velocity and technical debt to iteratively improve the pipeline.
• Technical Influence: Partnering with other engineers to drive initiatives that elevate best practices.
• Stakeholder Relationships: Building strong relationships with other teams to ensure they are able (and know how) to get the most benefit from the pipeline.

Our Data Stack...

• SQL and Python based pipelines built with dbt on Databricks
• Analysis via Python jupyter notebooks
• Pyspark in Databricks workflows for heavy lifting
• Streamlit and Python for dashboarding
• Airflow DAGs with Python for ETL
• Notion for data documentation