Sr Technical Lead

Summary

Highly skilled engineer with over 10 years of experience in software development and data engineering. Specializes in AWS Glue, PySpark, EMR, OOP, Docker, ECS, Spark Streaming, Kafka, Hudi/Iceberg, Glue Data Catalog, and Glue ETL. Proficient in designing and implementing scalable and efficient data pipelines, utilizing the latest technologies and best practices. Strong problem-solving skills and a proven track record of successfully delivering complex projects on time. Experienced and good understanding of building configurable platforms and frameworks is a big plus.

Skills

-Python, PySpark and Spark streaming: Mastery in Python, PySpark, and spark streaming. Expertise in leveraging PySpark to process and analyse large-scale high velocity datasets in a distributed computing environment. You will be involved in creating a near real-time stream processing framework and must be skilled in processing high-volume high-velocity time series data in real-time, enabling actionable insights and decision-making.

-Hudi/Iceberg: Proficient in working with advanced data storage formats like Apache Hudi or Apache Iceberg. Skilled in efficiently managing large-scale, incremental data sets with features such as data deduplication, ACID compliance, and schema evolution.

-EMR (Elastic MapReduce): Proficient in working with Amazon EMR, a cloud-based big data processing service. Skilled in designing and managing EMR clusters for running PySpark and other big data processing frameworks.

-Object-Oriented Programming (OOP): Strong understanding and implementation of OOP principles and design patterns. Proficient in developing high-quality, maintainable, and scalable code using OOP concepts.

-Kafka: Experienced in working with Apache Kafka, a distributed streaming platform. Skilled in building scalable and fault-tolerant data streaming pipelines using Kafka for real-time data ingestion and processing.

-ECS / Docker: Proficient in working with Amazon ECS, a scalable container orchestration service. Skilled in deploying, managing, and scaling Docker containers in ECS clusters, ensuring high availability and fault tolerance. Expertise in containerization using Docker, enabling the development and deployment of applications in a consistent and reproducible manner. Skilled in creating Docker images and managing containers for seamless application deployment.

-Glue : Expertise in utilizing the AWS Glue Data Catalog for efficient metadata management and data discovery. Skilled in creating and managing databases, tables, and schemas to organize and catalog data. Proficient in designing and implementing ETL (Extract, Transform, Load) processes using AWS Glue ETL. Skilled in defining and executing Glue workflows to transform and process data at scale. Extensive experience in using AWS Glue for data cataloguing and ETL operations. Proficient in designing and developing Glue jobs and workflows to process large datasets efficiently.

Big plus

-Terraform: Skilled in using Terraform for infrastructure provisioning and management. Experienced in writing Terraform scripts to create and manage AWS resources, ensuring infrastructure-as-code principles are followed.

-GitHub: Proficient in utilizing Git and GitHub for version control and collaborative software development. Experienced in managing repositories, branching, merging, and resolving conflicts effectively.

-Debezium: Experience with Debezium, an open-source change data capture platform. Proficient in capturing and streaming database changes in real-time, enabling reliable and scalable data integration.

Experience and personal traits

- Over 10 years of experience in software development and data engineering roles.

- Experience as a Java programmer in your early career is considered a pre

- Successfully delivered in multiple complex projects, preferably in an international context

- you have strong problem-solving skills with a focus on providing efficient and optimized solutions. Proven ability to identify and resolve complex technical issues in a timely manner.

- Well-versed in project management methodologies (agile scrum) and best practices.

- experienced in actively engaging with developers in the EU or US to drive collaboration and project success.

- You feel comfortable in an egalitarian non-hierarchical working environment, which means that you will not hold back to share your vision or opinion even if that means disagreement the more senior members of your team.

- Skilled in effectively communicating complex technical concepts and building strong relationships with team members.

Education

- Bachelor's or Master's degree in Computer Science, Software Engineering, or a related field.

- Relevant certifications in AWS, PySpark, Docker, Kafka, Hudi/Iceberg, Glue, or other related technologies are a plus