Staff Platform Engineer - Big Data Platform Management

CSIT develops products to advance the national security interests of Singapore. We use our products in a wide range of operations, including but not limited to Counter-terrorism and Computer Network Defence. We are looking for talented Senior Big Data Platform Staff Engineer. Join the awesome CSIT family and use cutting-edge technologies to protect the nation. 

Role

• We are seeking an experienced Big Data Platform Staff Engineer to join our team, responsible for managing and optimizing our Big Data infrastructure platform. The successful candidate will have a deep understanding of Big Data technologies, architectures, and systems, as well as experience with managing large-scale data processing and analytics environments. The role will involve designing, implementing, and maintaining the Big Data infrastructure platform, ensuring high availability, scalability, and performance.

Responsibilities

• Design, implement, and manage the Big Data infrastructure platform, including Hadoop, Spark, NoSQL databases, and data lakes.
• Manage Big Data clusters, including monitoring, troubleshooting, and optimizing cluster performance, as well as managing job scheduling and resource allocation.
• Design and implement storage solutions for Big Data workloads, including HDFS, S3, and other object storage systems.
• Ensure the security and governance of the Big Data infrastructure platform, including managing access controls, implementing data encryption, and conducting regular security audits.
• Collaborate with data engineers and data scientists to optimize Big Data application performance and provide technical support and guidance on using the Big Data infrastructure platform.
• Plan and manage Big Data resource capacity, including forecasting, procurement, and deployment of new hardware and software.

Requirements (Minimum Qualifications)

• Bachelor's degree in Computer Science, Engineering, or a related field.
• At least 8 years of experience in managing Big Data Platforms.  Proven experience with technologies like Spark, Kafka, Hadoop, Elastic Stack, Kudu, NiFi and Kafka.
• Strong knowledge of Big Data architectures, including data lakes, data warehouses, and data pipelines.
• Experience in troubleshooting and resolving issues of the Big Data stack, ranging from hardware, networking, operating system to software
• Strong understanding of data storage systems, including HDFS, S3, and other object storage systems.
• Experience with scripting languages, such as Python, Scala, or Java.
• Experience in managing physical and virtual infrastructures.
• Experience in supporting search workloads in multi-terabytes and billions of records.
• Experience in distributed systems.
• Proven experience in using DevOps tools and CI/CD pipelines effectively.
• Experience in leading engineering teams.

Nice to Have

• Certifications in Big Data, such as Hadoop Certified Developer or Spark Certified Developer.
• Experience with cloud-based Big Data platforms, such as AWS EMR, Google Cloud Dataproc, or Azure HDInsight.
• Experience with machine learning and deep learning frameworks, such as TensorFlow, PyTorch, or Scikit-learn.
• Familiarity with containerization, such as Docker or Kubernetes.
• Experience with agile development methodologies and version control systems, such as Git.
• Experience with cloud computing platforms, such as AWS, Azure, or Google Cloud.
• Experience with containerization, such as Docker or Kubernetes.

Why join us?

• The work is purposeful and meaningful 
• You will work with the best engineers 
• We work with modern technologies and tech stacks 
• We have excellent engineering culture and work-life balance 
• We aspire to engineering and operational excellence 
• We empower to innovate 
• We grow together as a family 

As CSIT is an agency under the Ministry of Defence (Singapore), only Singapore Citizens will be considered.