Data Engineer

Application deadline 27th Oct 2024

Position Title: Data Engineer

Reporting to: Amena Shakil

Job Group: 2/3

Location: Pakistan

Division: TSS Pakistan

Department/Unit:  Automations/Integration Hub

Position Related

A Data Engineer is responsible for designing, constructing, and maintaining scalable data pipelines, catalogues and architectures to support data-driven decision-making within the organization. The role involves transforming raw data into a format suitable for analysis and integrating it into data storage solutions.

A Data Engineer specializing in Python is responsible for building and managing data pipelines, transforming data into actionable insights, and ensuring data integrity across various platforms. This role requires a deep understanding of data architecture, ETL processes, and the ability to work with large datasets.

Goals

• Develop and maintain robust data pipelines to collect, process, and analyze large volumes of data from various sources.
• Design and implement data storage solutions, including data fabric, lakes and warehouses, ensuring optimal performance and security.
• Collaborate with data analysts to understand data needs and provide support for data access and analysis.
• Monitor data pipelines for performance and reliability, implementing necessary optimizations and troubleshooting issues as they arise.
• Ensure data quality and integrity by implementing data validation and cleansing processes.
• Document data architecture, pipeline processes, and data lineage to ensure clarity and compliance with best practices.
• Stay updated with industry trends and emerging technologies to continuously improve data engineering practices.
• Design and implement scalable data pipelines using Python to extract, transform, and load data from multiple sources.
• Monitor and troubleshoot data pipeline performance, implementing improvements as needed.

Eligibility Criteria

1. Education: bachelor’s or master’s degree in computer science, Information Technology, or a related field.

Experience / Skills

• 5-7 years of experience in data engineering, data architecture, or a related field.
• Proficiency in programming languages such as Python, Java, or Scala.
• Strong experience with data storage technologies such as SQL (MySQL, PostgreSQL) and NoSQL (MongoDB, Cassandra).
• Familiarity with ETL tools and data integration frameworks (e.g., Apache Spark, Apache NiFi, Airflow, Microsoft data fabric etc;).
• Knowledge of cloud platforms (AWS, Azure, GCP) and data-related services (e.g., AWS Redshift, Azure Data Lake).
• Understanding of data modeling, data warehousing concepts, and data governance.
• Excellent problem-solving skills and the ability to work independently and collaboratively.
• Strong communication skills to effectively collaborate with cross-functional teams.