Tech Lead/ Senior Architect - Big Data
Jakarta, Jakarta
NorthBay Solutions
Get access to all of our AWS best practices and insights. Subscribe Now AWS EXPERT We are a fast-growing, AWS focused onshore/offshore AWS Premier Partner NorthBay Solutions is proud to have earned the AWS competencies through...NorthBay Solutions, a premier US-based AWS partner with over a decade of experience serving clients globally, is seeking a passionate and experienced Tech Lead/Senior Architect to join our growing Data Engineering team.
About the Role: In this role, you will play a pivotal role in leading and architecting innovative solutions leveraging AWS Big Data technologies (e.g. S3, Redshift, EMR, Glue). You will collaborate closely with a global, multicultural team of Data Engineers and guiding project execution, ensuring technical excellence, and fostering a culture of continuous learning and supporting sales growth.
Key Qualifications & Required Knowledge:
- 10+ years of experience in technology, with a minimum of 6 years specifically focused on Data Engineering.
- Proven experience developing and implementing ETL pipelines on AWS that extract, transform, and load data for strategic business goals.
- Expertise in ETL Data Engineering using Python, Java, and Big Data technologies (Hadoop, PySpark, Spark SQL, Hive).
- Effective time management skills with a proven ability to manage and prioritize multiple project tasks.
- Experience creating and driving large-scale ETL pipelines in an AWS environment.
- Experience integrating data from diverse data sources.
- Strong software development and programming skills with a focus on data using Python/PySpark or Scala.
Preferred Qualifications (Nice to have):
- Experience with DevOps and CI/CD tools (Bitbucket, Bamboo, Ansible, Sonar, Atlassian suite).
- In-depth knowledge of core AWS services (IAM, CloudFormation, EC2, S3, EMR/Spark, Glue, Lambda, Athena, Redshift).
- Familiarity with popular data management tools (Data Lake, Databricks, Snowflake).
- Understanding of core data science concepts (statistics, modelling, algorithms).
- Experience with AWS AI/ML Services (SageMaker, Ground Truth) for building, training, and deploying machine learning models.
- Working knowledge of AWS infrastructure services (ECS, Cognito, CloudWatch, RDS: Postgres, API Gateway, Lambda) to support data engineering workflows and integrations.
- Acquainted with the value and evolution of Generative AI (GenAI) and its potential applications within the Big Data landscape.
- Basic familiarity with Amazon Q, a managed generative AI powered chat assistant, and Amazon Bedrock, a serverless service that hosts foundation models.
Technical Responsibilities:
- Design and develop innovative solutions leveraging Big Data technologies (Hadoop, Spark, etc.) to address complex data challenges.
- Architect and build highly scalable and distributed data systems on the AWS cloud platform.
- Design, develop, and manage robust ETL jobs and pipelines to ensure efficient data movement and transformation.
- Deploy production-ready data models, including data ingestion, configuration, resource management, and monitoring.
Leadership and Collaboration:
- Partner with sales to qualify opportunities, develop compelling technical solutions, and respond to RFPs.
- Serve as a thought leader, providing strategic guidance and mentoring junior team members as they leverage new data-driven solutions.
- Foster a culture of continuous learning and improvement by mentoring and training colleagues.
- Collaborate effectively to troubleshoot complex issues and provide timely guidance to team members.
Business Impact:
- Drive impactful business improvements by analysing data, identifying trends, and recommending process optimizations.
Additional qualities:
- Exceptional communication and English skills.
- Proven leadership.
- Client-facing expertise.
- Technological agility.
- Thrives in a fast-paced environment.
- Analytical & Detail-oriented.
- Self-driven leader.
- Team player.
* Salary range is an estimate based on our AI, ML, Data Science Salary Index 💰
Tags: Ansible APIs Athena AWS Big Data Bitbucket CI/CD CloudFormation Databricks Data management DevOps EC2 ECS Engineering ETL Generative AI Hadoop Java Lambda Machine Learning ML models Pipelines PostgreSQL PySpark Python Redshift SageMaker Scala Snowflake Spark SQL Statistics
Perks/benefits: Career development
More jobs like this
Explore more career opportunities
Find even more open roles below ordered by popularity of job title or skills/products/technologies used.