Lead Data Engineer

Company Description

Our Mission

Blend is an award-winning pure play data consultancy who help people do data right through project delivery across strategy and consulting, data science and BI, and data engineering. As a trusted Data & AI partner we co-create value with clients across a wide variety of industries. Our company has made the Inc. 5000 list of Fastest Growing Companies and currently have offices in Edinburgh, the Netherlands, the US, Uruguay and India. 

By combining our teams’ expert technical knowledge with a practical approach to value creation, we deliver outcomes that make a real change for our clients. From using computer vision to remotely monitor crops to implementing a BI dashboard to help swimmers win more medals – nothing we do is designed to be left on the shelf.

Job Description

Life as a Lead Data Engineer at Blend

We are looking for someone who is excited by the idea leading a data engineering squad to develop best in class analytical infrastructures and pipelines.

However, they also need to be aware of the practicalities of making a difference in the real world – whilst we love innovative advanced solutions, we also believe that sometimes a simple solution can have the most impact.

Our Lead Data Engineer is someone who feels most comfortable around solving problems, answering questions and proposing solutions. We place a high value on the ability to communicate and translate complex analytical thinking into non-technical and commercially oriented concepts, and experience working on difficult projects and/or with demanding stakeholders is always appreciated.

Reporting to the Director of Data Engineering the role will work closely with the other Data Engineering Leads and other capabilities in the organisation such as the Data Science, Data Strategy and Business Development teams.

This role will be responsible for driving high delivery standards and innovation within the data engineering team and the wider company. This involves delivering data solutions to support the provision of actionable insights for stakeholders.

Our Data Engineering Leads remain hands-on and work with the Data Engineers within their squad.

What can you expect from the role?

• Lead project delivery, covers overseeing the end-to-end delivery of projects and ensure robust project governance from a Data Engineering perspective.
• Squad management, responsible for managing a team of Data Engineers, from Junior to Senior levels, covering resource utilization, training, mentoring, and recruitment.
• Stakeholder engagement by preparing and present data-driven solutions to stakeholders, translating complex technical concepts into actionable insights.
• Data pipeline ownership by designing, develop, deploy, and maintain scalable, reliable, and efficient data pipelines.
• Domain expertise by keeping up to date on emerging trends and advancements within data ecosystems, ensuring the team remains at the forefront of innovation.
• Business development support by providing expert input into proposal submissions and business development initiatives from a Data Engineering perspective.
• Champion engineering excellence by promoting best practices for high-quality engineering standards that reduce future technical debt.
• Evolve best practices by continuously refining the Data Engineering team's ways of working, driving alignment across the squad and wider teams.
• Strategic contributions by collaborating with the Data Engineering Director on strategic workstreams, contributing to the Data Engineering strategy and Go-to-Market (GTM) initiatives.

Qualifications

What you need to have?

Experience:

• Proven experience in leading technical teams and building applications using microservice architecture.
• Extensive experience with Python and FastAPI.
• Strong understanding of microservices principles and components such as logging, monitoring, health checks, scalability, resilience, service discovery, API gateways, and error handling.

Technical Skills:

• Proficiency with Pydantic and data validation in FastAPI.
• Experience with containerization technologies (e.g., Docker, Kubernetes).
• Familiarity with CI/CD pipelines and tools.
• Knowledge of API design and implementation.
• Experience with monitoring and logging tools (e.g., Prometheus, Grafana, other).
• Knowledge of security best practices in microservices architecture.
• Familiarity with version control systems (e.g., Git) and collaborative development workflows.

Nice to Have:

• Experience working with Retriever models, including implementation of chunking strategies.
• Knowledge and use of vector databases.
• Understanding of optimal approaches in querying LLM models via API.
• Prompt engineering knowledge, with familiarity in different strategies.
• Exposure to various prompt engineering techniques in different scenarios.

Additional Information

All your information will be kept confidential according to EEO guidelines.