Data Engineer

Prudential’s purpose is to be partners for every life and protectors for every future. Our purpose encourages everything we do by creating a culture in which diversity is celebrated and inclusion assured, for our people, customers, and partners. We provide a platform for our people to do their best work and make an impact to the business, and we support our people’s career ambitions. We pledge to make Prudential a place where you can Connect, Grow, and Succeed.

Responsibilities:

• Current Context:
  • Design, develop, and maintain robust data pipelines for diverse sources using Qlik CDC, GCP Cloud Composer (Airflow), and Confluent Cloud Kafka.
  • Develop efficient SQL-based data transformations and create data models within BigQuery (lakehouse architecture), optimized for analytical and AI/ML use cases.
  • Implement data storage and processing solutions in GCP, adhering to data lakehouse principles and BigQuery PII best practices.
  • Build data pipelines to support AI/ML model development, including basic model prototyping.
  • Implement data quality checks and contribute to data lineage within our data governance framework.
  • Implement data platform monitoring and alerting using GCP Monitoring, Logging, and Cloud Composer.
  • Utilize Terraform, GitHub Actions and Jenkins for infrastructure deployment and CI/CD.
  • Implement secure data handling practices and customer-managed encryption keys where required.
  • Partner with cross-functional teams globally, communicating platform updates effectively.
• Role Briefly: Data Pipeline Development, Data Ingestion, Data ETL, Data Modeling, Data Lakehouse (BigQuery), Data Quality.
• Expectations for Three Months: Become familiar with our existing technology stacks, not only within your specific role but across the broader data platform ecosystem.
• Expectations Within One Year: Provide essential support to our AI and data platform initiatives, contributing to solutions that address resource constraints and enable advanced analytics. Specific contributions can be discussed.
   

Who We're Looking For:

• Non-Technical Skills & Mindset:
  • Impact-Driven & Results Focused:
    • Value-Oriented: Focused on delivering solutions that generate significant business value (millions USD impact).
    • Impact Conscious: Prioritizes work with the greatest technical and business impact. A focus on enabling data consumption through API creation is a plus.
  • Growth & Learning Mindset:
    • Cross-Functional Learner: Eager to learn and understand cross-functional knowledge beyond core expertise.
    • Technology Agnostic Learner: Willing to learn new technologies and adapt to evolving landscapes.
    • Efficient Learner: Able to leverage AI tools to maximize productivity and accelerate learning.
    • Best Practice Pragmatist: Loves to follow best practices but understands trade-offs and works around limitations when necessary. Demonstrated pro-activeness through contributions to open-source projects is highly valued.
  • Collaborative & Global Communicator:
    • Team Player: Collaborates effectively in global team environments. Adaptable and comfortable working within an Agile environment.
    • Excellent Communicator (English & Chinese): Fluent in both English and Chinese (Mandarin) to effectively communicate with global teams and stakeholders.
• Technical Concepts: We're looking for candidates with a strong grasp of:
  • Fundamental computer science knowledge
  • Root cause finding methodologies
  • Systematic/architectural thinking
  • Clean code/clean architecture principles and an aversion to over-design
• Technical Skills:
  • Python: Proficient in Python for data engineering and automation.
  • SQL: Expert in SQL, particularly for BigQuery and data lakehouse.
  • Cloud Development: Hands-on experience with GCP, including hybrid environments with on-premises DCs. Experience with AWS or Azure is also acceptable.
  • BigQuery PII: Experience with handling PII data and BigQuery PII features.

Tech Stacks:

• Compute & Hosting: GKE & GCE (RedHat), GCP Cloud Run & Cloud Functions
• Data Orchestration: GCP Cloud Composer (Airflow)
• Data Lakehouse: BigQuery
• Data Streaming: Kafka Ecosystem (Confluent Cloud, Debezium, Qlik)
• Monitoring & Observability: GCP Monitoring/Logging/Metrics, OpenTelemetry
• CI/CD: GitHub Actions, Jenkins
• Infrastructure as Code: Terraform
• Security: VPC SC & Policy Tags, Customer-Managed Encryption Keys (CMEK), Vault
• Containers: Docker, Kubernetes
• Data Governance: Collibra
• Data Visualization: Power BI

Prudential is an equal opportunity employer. We provide equality of opportunity of benefits for all who apply and who perform work for our organisation irrespective of sex, race, age, ethnic origin, educational, social and cultural background, marital status, pregnancy and maternity, religion or belief, disability or part-time / fixed-term work, or any other status protected by applicable law. We encourage the same standards from our recruitment and third-party suppliers taking into account the context of grade, job and location. We also allow for reasonable adjustments to support people with individual physical or mental health requirements.