Software Developer / Data Engineer

Title

Software Developer / Data Engineer

Job Description

We are looking for s strong passionate and skilled Data Engineer to join our MAC (Margin Assurance Cloud) project, to can help the team drive the project to fruition.

MAC is a bespoke in-house developed  AWS Serverless Cloud Native application and the core functionalities are:

• Data

  • Extract, load and transform data using familiar scripting language like SQL & Python for the purpose to produce automated, near real time Assurance Controls to the MA Team

• Self-Service

  • Access to Athena using workgroups & federated roles for end users (MA Team) to run their own queries for the purpose of investigations;

• Reporting & Alerting  

  • Capability to create reports from Athena views and send via email to the business customer;

  • Alerting capabilities & Monitoring on File Ingestion

Main Components (AWS Services used): S3, Athena, Lambda Functions, GLUE, EC2

Main Coding Language is: Python, PySpark and SQL

In support to our Clients  IT strategy we are in progress with a major migration project of the current on-prem Assurance system utilized by business Margin Assurance residing in FINOPS to the MAC Cloud environment.

The scope of the project build includes:

Data Ingestion and Transformation; S3 Loader Output Build; Athena SOX Validation Reports Build; Athena SOX Aggregation Reports Build;

Delivery will be an agile project which will be managed in monthly sprints. Altron will deliver the requirements as listed and prioritized in the backlog by the Vodacom Technical Delivery Lead and agreed to in each Sprint Planning Ceremony.

KEY RESPONSIBILITIES:

The Role:

• Implement scalable data pipelines and architectures using technologies like PySpark/Python/SQL

• Build out distributed data pipelines and compute tier that operates on AWS Lambda and Glue

• Serve as a technical resource for team members and mentor junior engineers

• Collaborate with team to deliver high-quality solutions that meet business requirements

• Ensure that code is well-designed, maintainable, and adheres to best practices and standards

• Play a key role in shaping the direction of engineering practices through working on a scrum-size team empowered to organize and ensure sprint deliverables are met as committed during sprint planning ceremonies

• Following the project development tools like JIRA, Confluence and GIT

• Assist DEVOPS Engineer in automation CICD practices.

• Evaluate and recommend new technologies and approaches to improve the performance, scalability, and reliability of our software systems

CORE RESPONSIBILITIES:

Skills / Requirements:

• 4 to 8 Years of Data Engineering or Software Development experience working on Data Driven eco systems

• Required to code complex transformations using loader specifications provided by the BA

• Be able to work with Big Data sets, very knowledgeable in understanding and solving data problems

• Be able to automate ingestion through building ingestion pipelines using Lambda or Glue

• Very skilled in coding languages PySpark/Python/SQL

• Must have worked on AWS, with the focus on the following services: S3, Athena, Lambda Functions, GLUE, EC2

• Experience in AWS and Certificate in AWS certification required

• Experience with data modelling and data architecture design required

QUALIFICATIONS, EXPERIENCE, & SKILLS:

Educational Qualifications:

• BSc Comp Sci/BEng

Professional Qualifications:

• AWS Professional Certification

Years of Experience:

• 4 to 8 Years of Data Engineering or Software Development experience working on a Data driven eco systems.

Education

Bachelor's Degree: Computer and Information Science (Required)

Languages

English