Sr Big Data Engineer (GCP)- Airflow and Oozie
India - Remote
Rackspace
As a cloud computing services pioneer, we deliver proven multicloud solutions across your apps, data, and security. Maximize the benefits of modern cloud.
About the Role:We are seeking a highly skilled and experienced Senior Big Data Engineer to join our dynamic team. The ideal candidate will have a strong background in developing and scaling both stream and batch processing systems, and a solid understanding of public cloud technologies, especially GCP. This role involves working in a remote environment, requiring excellent communication skills and the ability to solve complex problems independently and creatively.
What you will be doingBuild a reusable, and reliable code for stream and batch processing systems at scale. This includes working with technologies like Pub/Sub, Kafka, Kinesis, DataFlow, Flink, Hadoop, Pig, Hive, and Spark. Implementing automation/DevOps best practices for CI/CD, IaC, Containerization, etc.
What you will be doingBuild a reusable, and reliable code for stream and batch processing systems at scale. This includes working with technologies like Pub/Sub, Kafka, Kinesis, DataFlow, Flink, Hadoop, Pig, Hive, and Spark. Implementing automation/DevOps best practices for CI/CD, IaC, Containerization, etc.
Requirements
- About the Role:
- We are seeking a highly skilled and experienced Senior Big Data Engineer to join our dynamic team. The ideal candidate will have a strong background in developing batch processing systems, with extensive experience in Oozie, the Apache Hadoop ecosystem, Airflow, and a solid understanding of public cloud technologies, especially GCP. This role involves working in a remote environment, requiring excellent communication skills and the ability to solve complex problems independently and creatively.
- What you will be doing
- Develop scalable and robust code for batch processing systems. This includes working with technologies like Hadoop, Oozie, Pig, Hive, Map Reduce, Spark (Java), Python, Hbase
- Develop, Manage and optimize data workflows using Oozie and Airflow within the Apache Hadoop ecosystem
- Leverage GCP for scalable big data processing and storage solutions
- Implementing automation/DevOps best practices for CI/CD, IaC, etc.
- Requirements:
- Experience with GCP managed services and understanding of cloud-based batch processing systems are critical.
- Proficiency in Oozie, Airflow, Map Reduce, Java
- Strong programming skills with Java (specifically Spark), Python, Pig, and SQL
- Expertise in public cloud services, particularly in GCP.
- Proficiency in the Apache Hadoop ecosystem with Oozie, Pig, Hive, Map Reduce
- Familiarity with BigTable and Redis
- Experienced in Infrastructure and Applied DevOps principles in daily work. Utilize tools for continuous integration and continuous deployment (CI/CD), and Infrastructure as Code (IaC) like Terraform to automate and improve development and release processes.
- Ability to tackle complex challenges and devise effective solutions. Use critical thinking to approach problems from various angles and propose innovative solutions.
- Worked effectively in a remote setting, maintaining strong written and verbal communication skills. Collaborate with team members and stakeholders, ensuring clear understanding of technical requirements and project goals.
- Proven experience in engineering batch processing systems at scale.
- Hands-on experience in public cloud platforms, particularly GCP. Additional experience with other cloud technologies is advantageous.
* Salary range is an estimate based on our AI, ML, Data Science Salary Index 💰
Job stats:
0
0
0
Categories:
Big Data Jobs
Deep Learning Jobs
Engineering Jobs
Tags: Airflow Big Data Bigtable CI/CD Dataflow DevOps Engineering Flink GCP Hadoop HBase Java Kafka Kinesis Map Reduce Oozie Python Spark SQL Terraform
Regions:
Remote/Anywhere
Asia/Pacific
Country:
India
More jobs like this
Explore more career opportunities
Find even more open roles below ordered by popularity of job title or skills/products/technologies used.
Principal Data Engineer jobsData Engineer II jobsPrincipal Data Scientist jobsWriter - Freelance AI Tutor jobsContent writer - Freelance AI Tutor jobsData Manager jobsData Science Manager jobsData Scientist II jobsJunior Data Analyst jobsResearch Scientist jobsBusiness Data Analyst jobsSenior Machine Learning Researcher jobsBI Analyst jobsSr Data Engineer jobsSr. Data Scientist jobsSoftware Engineer, Machine Learning jobsData Science Intern jobsLead Data Analyst jobsBusiness Intelligence Engineer jobsJunior Data Engineer jobsJunior Data Scientist jobsData Analyst II jobsSenior AI Engineer jobsAzure Data Engineer jobsCopywriter - Freelance AI Tutor jobs
Data governance jobsSnowflake jobsLinux jobsHadoop jobsOpen Source jobsRDBMS jobsBanking jobsJavaScript jobsPhysics jobsMLOps jobsKafka jobsScala jobsData Warehousing jobsComputer Vision jobsNoSQL jobsGoogle Cloud jobsAirflow jobsSAS jobsOracle jobsData warehouse jobsLooker jobsKPIs jobsData Mining jobsPostgreSQL jobsStreaming jobs
R&D jobsClassification jobsCX jobsScikit-learn jobsGitHub jobsTerraform jobsScrum jobsDistributed Systems jobsPandas jobsPySpark jobsJira jobsIndustrial jobsBigQuery jobsRedshift jobsReact jobsRobotics jobsUnstructured data jobsJenkins jobsMySQL jobsMicroservices jobsdbt jobsMatlab jobsE-commerce jobsData strategy jobsPharma jobs