Data Engineer - Business Data Solution Group, Business Support Development Department (BSD)

Job Description:

Business Overview

Rakuten Group is based in Japan and serves about 100 million customers domestically and 1 billion customers globally. We offer over 70 services in various fields such as e-commerce, payment services, financial services, telecommunications, media, and sports.

Department Overview          

The Business Support Development Department is responsible for part of the RMS (ASP service) provided to stores by Rakuten Ichiba. Additionally, we develop and operate various applications that support the business staff who run Rakuten Ichiba. Our mission is to provide high-quality services using cutting-edge technology so that stores can efficiently expand their businesses.

The EC Business Data Analytics & Platform Section builds the data infrastructure that supports the massive transaction data of Rakuten Ichiba. We handle enormous data, including 50,000 stores, 300 million products, and an annual transaction volume exceeding 3 trillion yen, acting as a data hub to support business decision-making and data utilization. We also process Japan's largest traffic data during Rakuten Super SALE and other large events, striving to provide stable systems and continuously improve service quality.

Position:

Why We Hire

We are looking for new members to expand our services for both B2B (for stores) and B2C (for consumers) on Rakuten Ichiba. We use Hadoop and Google Cloud Platform for data storage and efficiently store and process data using Python, Spark, and Kafka.

This recruitment is for a data engineer responsible for expanding and optimizing the data pipeline that handles the massive transaction data of Rakuten Ichiba. This position plays a crucial role in providing data that supports important decision-making in Rakuten Ichiba, directly contributing to business growth and service improvement.

We are seeking an experienced lead software engineer. The ideal candidate will have a strong background in big data technology, cloud platforms, and data migration, with the ability to lead a team, manage projects, and provide high-quality solutions that promote operational efficiency and cost reduction.

Position Details

- Building Large-Scale Batch Data Pipelines: Use data processing frameworks and technologies like Spark, Hive, Kafka, Spark-Streaming, Hadoop, and Google Cloud Platform to build large-scale batch data pipelines. Utilize data pipelines to provide important decision-making data for data checks, customer acquisition, and other key business performance indicators in line with Rakuten Ichiba's guidelines.

- Data Pipeline Management: Build and maintain data pipelines for a large-scale e-commerce platform.

- Automation: Automate data ingestion processes to reduce manual data processing.

- Cloud Migration: Migrate on-premises data infrastructure to Google Cloud Platform.

- Application Management: Manage applications on the platform that ingest and process large amounts of data.

- Solving Complex Data Storage and Retrieval Issues: Resolve and clearly explain complex data storage and retrieval issues in big data pipelines.

- Debugging and Maintenance in Production: Debug and maintain applications in production using logging, alerting, and monitoring tools.

- Effective Collaboration: Collaborate effectively with other teams and cross-team members.

- Team Leadership: Lead a team of software engineers to achieve project goals.

Work Environment              

Development Environment

Python / Hadoop / Hive / Spark / Kafka / Google Cloud Platform

Mandatory Qualifications:

- Experience in developing and operating Hadoop, Spark, and Kafka applications

- Development experience in Python or Java/Scala

- Over 5 years of experience as a data engineer

- Development and operational experience with Google Cloud Platform or BigQuery

- Advanced knowledge of SQL and experience with relational databases

- Experience in building and optimizing data pipelines and datasets

- Experience in data modeling

- Development experience on Linux

- Strong analytical skills for data processing and transformation

- Project management skills to manage projects and provide high-quality solutions

- Team leadership experience in leading and mentoring software engineering teams

- Native-level proficiency in Japanese (reading, writing, speaking)

Desired Qualifications:

- Development experience using big data

- Experience in database operations and performance tuning

- Experience in infrastructure management with Docker, Kubernetes, etc.

- Deep understanding and experience with DevOps/Microservice architecture

- Operational experience using Jenkins or other CI technologies

- Experience in Scrum/Agile development

Others Information:

Additional information on Location

Osaka or Tokyo

Additional information on English Qualification

TOEIC score 800 or above

#engineer #datascientist  #researcher #commerce