Big Data Platform Operations Engineer

Brief Description

Reporting to DataOps Engineering Lead, the Big Data Platform Operations Engineer will be instrumental in designing, implementing, and maintaining scalable and reliable big data platforms to support the organization's data-driven initiatives. You will work closely with data engineers, data scientists, and software developers to architect and optimize big data infrastructure for storing, processing, and analyzing large volumes of data. This role requires expertise in big data technologies, distributed systems, and cloud platforms.  

Key Responsibilities

• Platform Architecture: Design and implement scalable and fault-tolerant big data platforms using distributed computing technologies such as Apache Hadoop, Apache Spark, or Apache Flink. Architect data storage solutions, including distributed file systems, NoSQL databases, and data warehouses. 
• Data Ingestion and Integration: Develop data ingestion pipelines to collect, process, and ingest data from various sources, including streaming data sources, databases, APIs, and log files. Implement ETL (Extract, Transform, Load) processes to preprocess and cleanse raw data for analysis. 
• Data Processing and Analysis: Optimize data processing workflows for efficiency, performance, and scalability. Develop and maintain data processing jobs, queries, and analytics workflows using distributed computing frameworks and query languages such as SQL, Hive, or Spark SQL. 
• Scalability and Performance: Design and implement strategies for scaling big data platforms to handle large volumes of data and diverse workloads. Optimize resource utilization, data partitioning, and parallel processing techniques to maximize performance and minimize latency. 
• Monitoring and Optimization: Develop monitoring and alerting solutions to track the health, performance, and availability of big data platforms. Implement automated scaling, load balancing, and resource management mechanisms to optimize platform utilization and performance. 
• Security and Governance: Ensure compliance with data governance policies, security requirements, and regulatory standards. Implement access controls, encryption, and auditing mechanisms to protect sensitive data and ensure data privacy and confidentiality. 
• Infrastructure as Code (IaC): Implement infrastructure automation using tools such as Terraform, Ansible, or CloudFormation. Define infrastructure configurations, provisioning scripts, and deployment pipelines to enable reproducible and consistent deployments of big data platforms. 
• Documentation and Training: Document platform architecture, configurations, and best practices. Provide training and support to data engineers and data scientists to ensure effective use of big data platforms and tools.

 

Qualifications

• Bachelor’s or master’s degree in computer science, Engineering, or related field.
• Solid understanding of big data technologies, distributed systems, and cloud computing principles.
• Proficiency in programming languages such as Python, Java, or Scala.
• Experience with big data frameworks such as Apache Hadoop, Apache Spark, or Apache Flink.
• Familiarity with cloud platforms such as AWS, GCP, or Azure. 
• System administration skills.
• Strong problem-solving skills and attention to detail.
• Excellent communication and collaboration skills.
• Ability to work independently and manage multiple priorities in a fast-paced environment.

 

How to Apply

If you feel that you are up to the challenge and possess the necessary qualifications and experience, kindly proceed to update your candidate profile on the recruitment portal and then click on the apply button. Remember to attach your resume.

 

We are the leading telecommunication company in East Africa. Our purpose is to transform lives by connecting people to people, people to opportunities and people to information. We keep over 42 million customers connected and play a critical role in the society, supporting over one million jobs both directly and indirectly while our total economic value was estimated at KES 362 Billion ($ 3.2 billion) for the 12 months through March 2021. We are listed on the Nairobi Securities Exchange (NSE) and with annual revenues of close to KES 298 Billion ($2.5 billion) as at March 2022. We were founded in 1997 as a fully owned subsidiary of Telkom Kenya before a 40 percent acquisition by Vodafone Group PLC in May 2000, and a public offering of 25 percent shares through the NSE in 2008. Under the management of Vodafone Group PLC, we welcomed Michael Joseph, as our first CEO, a few months later in July of 2000. He led the company’s growth to accommodate 16.71 million subscribers from the previous 20,000, largely owing to innovative products like M-PESA in 2007.