Data Engineer

• Data Pipeline Development: Design and develop scalable data pipelines using Apache Spark, ensuring efficient data processing and transformation.
• ETL Processes: Create and manage ETL (Extract, Transform, Load) workflows to integrate data from multiple sources into our data systems.
• Data Integration: Utilize Talend for data integration tasks, including data cleansing, transformation, and loading.
• Workflow Automation: Implement and manage data workflows using Apache Airflow to automate and optimize data processing tasks.
• Programming: Leverage Python to write scripts for data manipulation, transformation, and analysis.
• Database Management: Develop and optimize SQL queries for efficient data retrieval and analysis in relational databases.
• Collaboration: Work closely with data scientists, analysts, and other stakeholders to understand data needs and deliver effective solutions.
• Data Quality: Ensure data accuracy, consistency, and quality through rigorous validation and monitoring processes.
• Documentation: Create and maintain detailed documentation for data processes, pipelines, and integration workflows.
• Required Skills and Qualifications:
• Proficiency in Apache Spark for large-scale data processing and analytics.
• Experience with ETL processes and tools, including Talend.
• Familiarity with Apache Airflow for managing data workflows and automation.
• Strong programming skills in Python for data engineering tasks.
• Advanced SQL skills for querying and managing relational databases.

Requirements

• Minimum requirement at least 2 to 3 years experience in Talend (Must have) plus AWS and Glue