Big Data Engineer

Brief Description

Reporting to DataOps Engineering Lead. The Big Data Engineer will be responsible for designing, developing, and maintaining scalable big data solutions to enable efficient storage, processing, and analysis of large volumes of data. You will work closely with cross-functional teams including data scientists, data analysts, and software engineers to build and optimize data pipelines and systems. This role requires expertise in big data technologies, database management, and software engineering best practices.

 Key Responsibilities

• Data Pipeline Development: Design, implement, and maintain robust data pipelines for ingesting, processing, and transforming large volumes of structured and unstructured data. Develop ETL (Extract, Transform, Load) processes to cleanse, enrich, and aggregate data for analysis. 
• Data Storage Solutions: Architect and optimize data storage solutions, including distributed file systems, NoSQL databases, and data warehouses. Implement data partitioning, indexing, and compression techniques to maximize storage efficiency and performance. 
• Big Data Technologies: Utilize and optimize big data technologies and frameworks such as Apache Hadoop, Apache Spark, Apache Flink, and Apache Kafka. Develop and maintain data processing jobs, queries, and analytics workflows using distributed computing frameworks and query languages. 
• Scalability and Performance: Optimize data processing workflows for scalability, performance, and reliability. Implement parallel processing, distributed computing, and caching mechanisms to handle large-scale data processing workloads. 
• Monitoring and Optimization: Develop monitoring and alerting solutions to track the health, performance, and availability of big data systems. Implement automated scaling, load balancing, and resource management mechanisms to optimize system utilization and performance. 
• Data Quality and Governance: Ensure data quality and integrity throughout the data lifecycle. Implement data validation, cleansing, and enrichment processes to maintain high-quality data. Ensure compliance with data governance policies and regulatory standards. 
• Collaboration and Documentation: Collaborate with cross-functional teams to understand data requirements and business objectives. Document data pipelines, system architecture, and best practices. Provide training and support to stakeholders on data engineering tools and technologies.

 

Qualifications

• Bachelor's or master’s degree in computer science, Engineering, or related field. 
• Proven professional SQL capabilities
• Solid understanding of big data technologies, distributed systems, and database management principles.
• Proficiency in programming languages such as Python, Java, or Scala.
• Experience with big data frameworks such as Apache Hadoop, Apache Spark, or Apache Flink.
• Knowledge of database systems such as SQL databases, NoSQL databases, and distributed file systems.
• Familiarity with cloud platforms such as AWS, GCP, or Azure.
• Strong problem-solving skills and attention to detail.
• Excellent communication and collaboration skills.
• Ability to work independently and manage multiple priorities in a fast-paced environment.

 

How to Apply

If you feel that you are up to the challenge and possess the necessary qualifications and experience, kindly proceed to update your candidate profile on the recruitment portal and then click on the apply button. Remember to attach your resume.

We are the leading telecommunication company in East Africa. Our purpose is to transform lives by connecting people to people, people to opportunities and people to information. We keep over 42 million customers connected and play a critical role in the society, supporting over one million jobs both directly and indirectly while our total economic value was estimated at KES 362 Billion ($ 3.2 billion) for the 12 months through March 2021. We are listed on the Nairobi Securities Exchange (NSE) and with annual revenues of close to KES 298 Billion ($2.5 billion) as at March 2022. We were founded in 1997 as a fully owned subsidiary of Telkom Kenya before a 40 percent acquisition by Vodafone Group PLC in May 2000, and a public offering of 25 percent shares through the NSE in 2008. Under the management of Vodafone Group PLC, we welcomed Michael Joseph, as our first CEO, a few months later in July of 2000. He led the company’s growth to accommodate 16.71 million subscribers from the previous 20,000, largely owing to innovative products like M-PESA in 2007.