Senior Data Engineer

As the living, growing home of our national story, The National Archives is already a special place to work. We’re an institution nearly 200 years old with a collection spanning 1,000 years of history. But it’s where we go next that makes things really interesting. 

In our strategic vision: Archives for Everyone, we set ourselves the challenge of becoming the 21st Century national archive - a different kind of cultural and heritage institution: Inclusive, Entrepreneurial, Disruptive. We won’t become this overnight. It will take time, focus, effort and daring.

That’s where you come in. Because we can’t do this without you.

Job Overview

Salary: £60k (£55,549 + £4,451 market supplement)
Grade: F / Senior Executive Officer
Contract Type: Permanent
Closing date: Monday 21st April 2025 at midnight

Shape the Future of Digital Archives
At The National Archives, we don’t just record history—we transform how it’s preserved, accessed, and understood. We are the guardians of the UK’s digital Public Records, ensuring that government decisions, events, and societal moments are accessible for generations to come. Digital records aren’t just files on a server; they are our collective memory, shaping the future of research, democracy, and public engagement.

As a Senior Data Engineer, you’ll play a crucial role in designing, building, and optimising the data infrastructure that powers our digital archive. You will engineer scalable, secure, and resilient data pipelines that ensure public records remain structured, searchable, and reusable for researchers, educators, and policymakers.

This is not just a technical role—it’s an opportunity to influence how the UK’s most valuable digital assets are accessed. You’ll be working with cutting edge technology to solve some of the most complex and meaningful data engineering challenges in the public sector.

Why Join Us?
• Your work will ensure that raw government records are transformed into a structured, accessible digital archive that empowers researchers, educators, and the public.
• We embrace innovation, leveraging AWS, scalable data pipelines, machine learning, and graph databases to tackle unique challenges.
• Work alongside data engineers, digital records specialists, designers and developers who are passionate about shaping the future of digital archiving.
• Whether it's technical, leadership training or mentoring opportunities, we invest in your career development.
• Based in an iconic riverside location on the banks of the Thames, with access to an onsite gym and green spaces for the perfect balance of work and outdoor time.


What You’ll Be Doing
Engineering Data Platforms
• Designing, implementing, and optimising scalable data platforms and pipelines that support digital archiving.
• Automating data cleansing, validation, standardisation, migration, and transformation, ensuring high-quality and structured data delivery.
• Engineering secure and reliable data processes that meet the needs of diverse user groups, from government departments to academic researchers.
• Managing data migrations across platforms, ensuring data integrity and security throughout the process.
• Advocating for best practices in data governance, ensuring data remains secure, structured, and reusable.
Who We’re Looking For
• Proven hands-on experience with large-scale data querying, transformation, and integration using tools like Talend, Python, SQL, SPARQL, and Shell scripting.
• Strong knowledge of data models and structures (JSON, XML, CSV, RDF, relational and graph databases).
• Experience with problem-solving complex data challenges, balancing innovation with practical implementation.
• Strong collaboration and stakeholder engagement skills, capable of working across teams to influence data strategy and best practices.

Application Process
• Interview: Interviews will be held on-site and will include a technical test.
• Personal Statement:
We value authenticity and personal insight—use this as an opportunity to tell us what excites you about working with data at scale and tackling complex challenges at The National Archives. We ask all applicants to submit CV details and a 750-word personal statement, answering the following:

What skills and experience would you bring to The National Archives?
What skills would you like to develop in this role, and how do you see yourself growing with us?

Selection for interview will be based on the ‘essential’ requirements in the job description so please ensure that you demonstrate how you meet these requirements within your statement.

We understand that you might use artificial intelligence (AI) and other resources to assist with your application; however, please ensure all information you provide is factually accurate, truthful, and original and doesn’t include ideas or work that isn’t your own. We encourage you to showcase your unique knowledge and skills using your own voice.

SC clearance/willingness to obtain SC clearance will be required for this role. This requires candidates to have been resident in the UK for at least the past three years. Please do not apply if you have been resident in the UK for less than three years as your application will be rejected.

Please note: Requests for flexible working will be considered.

Job Description

Roles and Responsibilities

Engineer Data Platforms:

• Ensure the integrity, security and availability of the organisation’s data and data services.
• Design, implement, and enhance data platforms and infrastructure, including ETL tools such as Talend, bespoke software, AWS messaging architectures.
• Build and operate scalable, accurate data processes to meet the needs of the business and users.
• Automate data processing tasks such as cleansing, validation, standardisation, migration and transformation to deliver high-quality, structured data to operational teams.
• Design and build the data transformation pipeline to collect, process and route data between loosely integrated systems and services.  Ensuring secure, fast, reliable, good value for money in line with the needs of our core digital archiving services.
• Create and refine data models, schemas and ontologies tailored to The National Archives business needs, in consultation with the business.
• Manage data migration (structured data, unstructured data, documents, and digital objects) across platforms. Maintaining data integrity throughout the migration process.
• Stay informed about sector trends and new technologies. Develop your expertise in relevant tools, technologies and infrastructure platforms to continuously improve data services

Collaborate and deliver technical consultancy

• Embed good data engineering practices across the department and directorate. Championing data governance and addressing risks to data integrity.
• Share your work openly by leading ‘show and tells’, blog posts, papers, presentations and through making your designs and code available for re-use whenever possible.
• Apply your coaching and mentoring skills to support the professional development of less experienced colleagues, including junior staff and apprentices. Line-manage staff and contractors as necessary.
• Build strong relationships across TNA and the wider archives sector to advise and influence archivists, subject specialists and senior staff responsible for archival data standards, schemas and policies.
• Develop your knowledge of archival practices and related standards and approaches.
• Work closely with multi-disciplinary Agile software development team(s). Advocate for and champion good data engineering practices.

Working Conditions

• Hybrid working is available. Staff are expected to spend at least 60% of their time on-site. This may be higher when staff are new to a role or for staff in trainee roles.
• Normal office environment
• Display Screen Equipment user
• May come into contact with distressing material

Person Specification

Essential skills and experience

*Technical Expertise:

Proven hands-on experience data engineering technologies including:

• Large-scale data querying, analysis and manipulation for structured and textual data. For example: ETL tools and pipelines (including Talend), NLP and entity extraction, Python, Shell scripting, XSLT, SPARQL, SQL, Cypher.
• Data encoded in different formats and structures. For example: JSON, XML, CSV, RDF, relational data.
• Deep understanding or relational, graph and document database models with the ability to choose the right approach for each use case

*Problem Solving and Delivery:

Strong track record of resolving complex challenges, developing strategic solutions, including conducting or commissioning technical research and overcoming technology constraints to enable effective service delivery.

Data Driven Development:

• Skilled in advocating for a data-driven approaches that are secure, scalable and user-focused in a rapidly evolving environment.
• Familiarity with conducting technical spikes and developing prototypes to model scenarios, demonstrate ideas, evaluate options and identify creative solutions to unique challenges.

*Collaboration:

• Excellent collaboration skills, able to build productive, trust-based working relationships with across teams and work effectively with subject specialists.
• Strong communication and networking skills, with the presence and confidence to represent The National Archives internally and externally to build relationships, exchange knowledge, advocate and influence stakeholders, suppliers and partners.

Desirable

• Experience of working in the AWS environment with knowledge of technical design patterns, services and infrastructure relevant to data processing operations.
• Familiarity of Agile methodologies and contributing effectively with multidisciplinary teams.
• An understanding of the value of records and archives. Familiarity with the relevant data standards and approaches.
• Experience of working with probabilistic, uncertain or messy historical data with an understanding of relevant statistical techniques.

Other Requirements

• SC level Security Clearance or willingness to undergo SC clearance.
• Resident in the UK for at least the last three years.
• Applicants must have the right to work in the UK.

The Civil Service is committed to attract, retain and invest in talent wherever it is
found. To learn more please see the Civil Service People Plan and the Civil Service
D&I Strategy.

Benefits 

 

Generous benefits package, including pension, sports and social club facilities, onsite gym, discounted rates at our on-site cafe and opportunities for training and development.  Annual leave entitlement of 22 days per calendar year (rising to 25 after the first year, and incrementally to 30 days after six years) and 10½ days public and privilege holidays per annum.

 

Any move to The National Archives from another employer will mean you can no longer access childcare vouchers. This includes moves between government departments. You may however be eligible for other government schemes, including Tax-Free Childcare. Determine your eligibility at  https://www.childcarechoices.gov.uk/ (opens in new window)

Reasonable adjustments

 

If a person with disabilities is put at a substantial disadvantage compared to a non-disabled person, we have a duty to make reasonable changes to our processes. 

If you need a change to be made so that you can make your application, you should: 

• Contact The National Archives via careers@nationalarchives.gov.uk as soon as possible before the closing date to discuss your needs 

• Complete the ‘Reasonable Adjustments’ section of your application form to tell us what changes or help you might need further on in the recruitment process. For instance, you may need wheelchair access at interview, or if you’re deaf, a Language Service Professional

​

Feedback will only be provided if you attend an interview or assessment.  

 

Security 

 

Successful candidates must pass a disclosure and barring security check. 

People working with government assets must complete basic personnel security standard checks (opens in new window)

 

Nationality requirements 

 

This job is broadly open to the following groups: 

• UK nationals

• nationals of the Republic of Ireland

• nationals of Commonwealth countries who have the right to work in the UK

• nationals of the EU, Switzerland, Norway, Iceland or Liechtenstein and family members of those nationalities with settled or pre-settled status under the European Union Settlement Scheme (EUSS) (opens in a new window)

• nationals of the EU, Switzerland, Norway, Iceland or Liechtenstein and family members of those nationalities who have made a valid application for settled or pre-settled status under the European Union Settlement Scheme (EUSS)

• individuals with limited leave to remain or indefinite leave to remain who were eligible to apply for EUSS on or before 31 December 2020

• Turkish nationals, and certain family members of Turkish nationals, who have accrued the right to work in the Civil Service

Further information on nationality requirements (opens in new window)

 

Working for the Civil Service

 

The Civil Service Code  (opens in new window) sets out the standards of behaviour expected of civil servants. 
 
We recruit by merit on the basis of fair and open competition, as outlined in the Civil Service Commission's recruitment principles. (opens in new window) 

 

The Civil Service embraces diversity and promotes equal opportunities. As such, we run a Disability Confident Scheme (DCS) for candidates with disabilities who meet the minimum selection criteria. 

The Civil Service also offers a Redeployment Interview Scheme to civil servants who are at risk of redundancy, and who meet the minimum requirements for the advertised vacancy. 

 

This vacancy is part of the Great Place to Work for Veterans initiative. (opens in new window)

 

Contact point for applicants:

• Name: The National Archives Recruitment Team 

• Email: careers@nationalarchives.gov.uk 

 

Further information 

 

If you feel your application has not been treated in accordance with the Recruitment Principles and you wish to make a complaint, in the first instance, you should contact The National Archives via email: careers@nationalarchives.gov.uk If you are not satisfied with the response you receive from the Department, you can contact the Civil Service Commission at https://civilservicecommission.independent.gov.uk/recruitment/recruitment-complaints/ (opens in new window)