Senior Data Engineer -AI & DATA Office (RMI AI & Data Div)

Job Description:

About Organization

Our mission is to empower our organization with AI-driven solutions that drive excellence and competitive advantage.

Identifying and implementing AI use cases across all departments to optimize processes and drive value.
Ensuring ethical and responsible AI practices are at the core of our AI platform's development and deployment.
Sharing our AI expertise with our industry peers, contributing to the broader AI community, and staying at the forefront of AI advancements.

We aim to position ourselves as leaders in the AI landscape, delivering tangible benefits to our organization, employees, and customers.

Job Duties

• Drive the strategy and execution of Rakuten Mobile's data engineering roadmap, aligning with the company’s goals and industry best practices

• Design and implement high-availability and fault-tolerant data architectures capable of handling complex, distributed data systems

• Define, guide, and refine architecture for data processing, pipeline design, and data storage solutions to support advanced analytics and real-time decision-making

• Oversee the design, development, and optimization of large-scale ETL and ELT pipelines

• Ensure seamless integration of diverse data sources using modern tools and technologies, enhancing data ecosystem efficiency and robustness

• Lead, mentor, and develop a high-performing team of data engineers, fostering collaboration and continuous learning

• Act as a technical leader, promoting best practices in data engineering, including coding standards, quality checks, and performance tuning

• Supervise root cause analysis and troubleshoot production issues to ensure data accuracy, availability, and performance

• Implement improvements based on technical incident post-mortems and lessons learned

• Conduct research and comparative analysis on emerging data technologies to ensure Rakuten Mobile stays at the forefront of innovation

• Recommend tools, frameworks, and methodologies that enhance data engineering efficiency, performance, and scalability

• Ensure comprehensive documentation of data architecture, integrations, and pipelines for operational transparency and knowledge sharing

• Champion data governance, compliance, and security best practices across the team and organization

• Implement data pipeline monitoring and alerting systems to proactively identify, resolve issues and Optimize data pipelines for performance and cost-efficiency.

• Develop and maintain a comprehensive data catalog to ensure data discoverability and usability by Implementing data lineage tracking to provide visibility into data flow and transformations.

• Utilize advanced analytics and machine learning techniques to optimize data processing workflows.

• Implement caching and indexing strategies to improve data retrieval times.

• Create and maintain a knowledge base of best practices and lessons learned in data engineering.

Minimum Qualifications

• 12-20 years of experience in data engineering, software engineering, or related fields, with a proven track record in large-scale, high-volume data environments

• Prior experience in a senior or head-level data engineering role, with demonstrated leadership capabilities

• Mastery of NiFi, Apache Spark, and distributed data ecosystems

• Strong expertise in Java and Scala programming languages for data applications

• In-depth experience with NoSQL databases, Presto, NiFi, Airflow, and complex SQL queries

• Proficiency in Data Modeling for both SQL and NoSQL data stores

• Hands-on experience in Kubernetes application development and K8s-centric applications

• Experience with ETL pipeline development in public cloud environments (AWS, Azure, Google Cloud) is a strong advantage

• Knowledge of POSTGIS and GeoSpark for geospatial data processing is beneficial

• Familiarity with Spring Framework for web application development

• Exceptional problem-solving abilities, capable of addressing complex technical challenges with a proactive, solution-oriented mindset

• A collaborative team player with the ability to communicate complex technical information to both technical and non-technical stakeholders

• Skilled in cross-functional collaboration, able to work effectively across departments and with senior leadership

Languages: