Lead Data Engineer – Core Data Platform

We are seeking an experienced Lead Data Engineer– Core Data Platform to drive the development and stability of our cloud-native data infrastructure. This role is crucial in ensuring our data platform is secure, reliable, and scalable. You’ll be responsible for the provisioning and automation of key components including Databricks, Airflow (Astronomer), GitLab CI/CD, Terraform, and monitoring tools such as Grafana. You will collaborate closely with DevOps, Security, and Architecture teams and provide platform support to other data teams, including those building the new Data Warehouse in Databricks. As a hands-on technical leader, you’ll contribute to the platform codebase while defining standards, improving automation, and ensuring operational excellence across environments.

• Design and maintain the core data platform infrastructure including Databricks, Airflow (Astronomer), AWS, CI/CD pipelines, and monitoring. 

• Manage and provision cloud environments using Terraform; maintain GitLab-based CI/CD workflows. 

• Own the RBAC implementation and environment governance ensuring secure and maintainable access controls. 

• Monitor cost and performance metrics using observability tools 

• Troubleshoot infrastructure-level issues and support DataOps teams working on the Data Lakehouse. 

• Collaborate closely with DevOps, Security, Architecture, and Product teams to align on platform standards. 

• Support deployment processes from development to production for data workflows. 

• Maintain documentation of architectural decisions, operational procedures, and tooling. 

• Proactively identify and implement automation and improvements across platform components. 

• Contribute directly to platform development, including coding, reviews, and mentoring. 

• Maintain a modular and scalable Terraform repository structure for multi-environment deployments. 

• Contribute to the development of internal tooling for platform automation and efficiency. 

• Define tagging strategies and cost monitoring standards across environments and workspaces. 

• Coordinate incident response and platform stability improvements. 

• Partner with Security and Cloud Governance teams on policies, audits, and compliance initiatives. 

• Track and address technical debt within platform infrastructure components. 

• Minimum 6 years of experience in platform or DevOps engineering in a data context. 

• Expertise in provisioning and automating cloud infrastructure using Terraform. 

• Strong knowledge of AWS services (IAM, networking, S3, cost tracking) and Databricks. 

• Handson experience with orchestrators like Airflow (Astronomer) and CI/CD pipelines (GitLab). 

• Familiarity with Databricks workspace and admin configurations. 

• Experience implementing RBAC, securing access and environments. 

• Experience in monitoring performance and setting up alerting/observability (Grafana, Prometheus). 

• Strong coding skills in Python and pySpark. 

• Ability to work closely with both technical and business stakeholders. 

• Mindset focused on scalability, reliability, and documentation. 

• Deep understanding of CI/CD strategies for data platforms and workflow-driven applications. 

• Familiarity with observability stacks (logs, metrics, traces) and SRE practices. 

• Experience in supporting multi-account cloud environments and cross-region deployments. 

• Proficiency in debugging infrastructure-related issues in cloud-native data pipelines. 

• Strong documentation habits and ability to produce clear, actionable technical runbooks. 

• Familiarity with platform cost optimization tools and practices (e.g., AWS Cost Explorer, native tagging, budgeting). 

With a fleet of 287 modern container ships and a Vessel Capacity 2.2 million TEU, as well as a Container Capacity 3.2 million TEU including one of the world’s largest and most modern reefer container fleets, Hapag-Lloyd is one of the world’s leading liner shipping companies. In the Liner Shipping segment, the Company has around 13.500 employees and 400 offices in 139 countries. Hapag-Lloyd has a container capacity of 11.9 million TEU – including one of the largest and most modern fleets of reefer containers. A total of 114 liner services worldwide ensure fast and reliable connections between more than 600 ports across the world. In the Terminal & Infrastructure segment, Hapag-Lloyd has stakes in 20 terminals in Europe, Latin America, the United States, India, and North Africa. The roughly 2.600 employees assigned to the Terminal & Infrastructure segment deal with terminal-related activities and provide complementary logistics services at selected locations.