Manager, Data Architecture & Engineering

Purpose of the job

Oversee and lead a team of data engineers and modelers to design, implement and maintain robust data models and pipelines. The role is critical for data-driven decision making within the organization and ensuring the integrity and availbility of data for various business functions.

Also responsible for overseeing, and guiding the development, integration, testing and maintenance of existing and new applications using a modern data stack. The solutions delivered need to be resilient, scalable, secured and with high performance that meet all the functional and non-functional requirements.

Duties and responsibilities

• Identifying and documenting big data use case requirements

• Work with implementation teams from concept to operations, leading the design and engineering of the technical solutions;

• Analyze and resolve medium to moderately large risk production problems, assess alternatives and implement long-term solutions

• Guide the Data and Insights team to develop quality Hive based ELT software and continuously improve the development process.

• Assist in designing a new revolutionary Meta Data Driven Data Ingestion system based on Hadoop technologies to support Reporting and Analytics for business needs.

• Participate in code reviews to ensure solution meets requirements and identify additional use cases.

• Provide data expertise and guidance to data analysts focused on providing self-serve analytic systems.

• Design and code data processes.

• Envision and create data integration solutions that meet the requirements of the business.

• Work on the creation of a flexible and sustainable Data Ingestion process to load data into our Hadoop based Data Lake.

• Work on Data Modeling to serve current ODS, OLAP and Data Warehouse data stores at the conceptual, logical and physical models.

• Detect and solve issues effectively.

Strategy & Planning

• Develop data modeling and engineering strategies aligned with the organization's business objectives.

• Collaborate with stakeholders to understand their data requirements and translate them into actionable plans.

• Identify opportunities for data-driven improvements and efficiency within existing systems.

Data Modeling

• Lead the design and implementation of data models that ensure data accuracy, consistency, and integrity.

• Define data structures and schemas to facilitate data storage, retrieval, and analysis.

• Optimize data models to support business intelligence, reporting, and analytics needs.

Data Engineering

• Manage the development and maintenance of data pipelines, ETL (Extract, Transform, Load) processes, and data integration solutions.

• Ensure data quality and consistency across different data sources and systems.

• Stay up-to-date with the latest data engineering technologies and best practices.

Team Leadership & Management

• Recruit, train, and mentor a high-performing team of data engineers and analysts.

• Provide guidance and support to team members, fostering their professional growth and development.

• Set clear objectives, monitor performance, and conduct regular performance reviews.

Collaboration 

• Collaborate with cross-functional teams, such as data scientists, software developers, and business analysts, to ensure seamless data flow and utilization.

• Work closely with IT teams to ensure the availability and scalability of data infrastructure.

Data Governance and Security

• Ensure compliance with data governance policies and industry regulations.

• Implement security measures to safeguard sensitive data and maintain data privacy. 

Problem Solving & Troubleshooting

• Identify and resolve data-related issues promptly, ensuring minimal disruption to business operations.

• Troubleshoot data quality and performance problems, implementing appropriate solutions.

Reporting & Documentation

• Prepare and present regular reports on data modeling and engineering activities, progress, and results to senior management.

• Maintain comprehensive documentation of data models, processes, and architecture.

Job specification

Education

• Bachelor degree of Engineering in Computer Systems, or Computer Science.
• Masters degree in Software Development or Data Science is a plus.

Experience

• At least 8 years of overall relevant IT experience
• At least 4 years of experience with Big Data projects (using Hadoop, SQL, ETL, and/or similar technologies.
• Hands on delivery experience working on popular Hadoop distribution platforms like Cloudera or Horton Works
• Extensive hands on experience in Hadoop eco-system products MapReduce, Hive, Spark, Sqoop, Flume, Oozie , HBase and Kafka
• Coordinating the movement of data from original data sources into noSQL data lakes and cloud environments
• Hands-on experience with tools used in conjunction of Hadoop MapReduce/Spark/Hive.
• Source control (preferably Git based)
• Knowledge of agile development methodologies, mainly scrum
• Experience in relevant IDE framework
• Good experience on ETL Technologies and concepts of Data Warehouse
• Proficient in a programming language like Java or Python or Scala
• Working knowledge and experience in Unix environment & experience in shell scripting
• RDBMS background, must be SQL conversant (SQL Server/Oracle and/or DB2)
• Experienced in importing and exporting the data using Sqoop from HDFS to Relational Database systems and vice-versa
• Performance tuning, troubleshooting data-related technical issues and identify proper solutions
• Knowledge of workflow/schedulers like Oozie
• Solid understanding of different file formats and data serialization formats such as ProtoBuf, Avro, JSON
• Experience in ETL processing involving large volume of data Ingestion into Data lake with utilities around Bigdata Platform

Skills and abilities

• Strong ability to drive complex technical solutions deployed at an enterprise level; ability to drive big data technology adoption and changes through education and partnership with stakeholders
• Demonstrated experience in working with the vendor(s) and user communities to research and test new technologies to enhance the technical capabilities of existing Hadoop cluster
• Ability to negotiate, resolve, and prioritize complex issues and provide explanations and information to others on difficult issues, assess alternatives and implement long-term solutions
• Self-starter who can work with minimal guidance
• Strong communication skills
• Very good English both written & spoken.