Senior Data Engineer

Our Purpose

Mastercard powers economies and empowers people in 200+ countries and territories worldwide. Together with our customers, we’re helping build a sustainable economy where everyone can prosper. We support a wide range of digital payments choices, making transactions secure, simple, smart and accessible. Our technology and innovation, partnerships and networks combine to deliver a unique set of products and services that help people, businesses and governments realize their greatest potential.

Title and Summary

Senior Data Engineer

Role Overview
As a Senior Data Engineer, you will work on the architecture, development, and optimization of data pipelines and lakehouse solutions on Databricks (AWS). This role requires a strong focus on data governance, security, lineage, and performance, in line with Open Banking regulatory requirements. You’ll be working closely with data platform teams, compliance officers, and product stakeholders to deliver clean, secure, and highly available data solutions.


Key Responsibilities
- Databricks Platform Engineering (AWS)
Design and implement Lakehouse architecture using Databricks, AWS S3, Glue Catalog, and Delta Lake.
Set up and manage multi-environment Databricks Workspaces (Dev, UAT, Prod) with consistent configuration and governance.
Implement cluster policies for standardized compute usage and cost efficiency.
Create and orchestrate Databricks Jobs and Workflows, integrating with CI/CD systems and version control.

- Security & Governance for Open Banking
Configure fine-grained access controls using Unity Catalog, Lake Formation, and IAM roles.
Manage user provisioning and role-based access with SCIM integrations to IdPs (e.g., Okta, AWS SSO).
Implement audit trails, lineage tracking, and data masking techniques to meet GDPR, PSD2, and Open Banking standards.
Collaborate with compliance teams to align data handling practices with consent and privacy regulations.

- Data Engineering & Pipelines
Develop scalable ETL/ELT pipelines in PySpark and Spark SQL, handling batch and streaming data flows.
Ingest and transform data from Open Banking APIs, Amazon Aurora, and third-party aggregators.
Optimize pipeline performance using Z-Ordering, Delta Lake compaction, and caching strategies.

- Monitoring, Support & Troubleshooting
Monitor cluster health, job failures, and performance metrics using native tools and AWS CloudWatch.
Diagnose and resolve issues across jobs, notebooks, and integrations with AWS services.
Act as the go-to Databricks expert for other engineering and analytics teams.

- Best Practices & Reusability
Enforce coding standards for modular notebooks, library packages, and secure secrets handling, use of parameterized, reusable notebooks, schema enforcement with Delta Lake, designing segregated compute clusters
Promote reusability, version control, and deployment automation using GitHub and CI/CD pipelines.


Must-have skills
6+ years of hands-on Databricks experience on AWS, with a focus on scalable data pipeline development.
Proficiency in PySpark, Spark SQL, Delta Lake, and Databricks Workflows.
Strong understanding of cloud data lake architecture using S3, Glue, and Lake Formation.
Hands-on experience with user access management, IAM policies, and Unity Catalog or similar governance tools.


Good-to-Have Skills
Experience with ETL tools such as HVR, AWS Glue or DBT.
Familiarity with Amazon Aurora (MySQL/PostgreSQL) or equivalent relational databases.
Understanding of Open Banking APIs, API standards (e.g., FAPI), and consent frameworks.
Exposure to DevOps practices and tools like Terraform, GitLab, Jenkins.
Experience working in a regulated industry, preferably Open Banking, fintech, or financial services.


Preferred Certifications
Databricks Certified Data Engineer – Associate or Professional
AWS Certified Solutions Architect – Associate

Corporate Security Responsibility

All activities involving access to Mastercard assets, information, and networks comes with an inherent risk to the organization and, therefore, it is expected that every person working for, or on behalf of, Mastercard is responsible for information security and must:

• Abide by Mastercard’s security policies and practices;

• Ensure the confidentiality and integrity of the information being accessed;

• Report any suspected information security violation or breach, and

• Complete all periodic mandatory security trainings in accordance with Mastercard’s guidelines.