Data Engineer

This role is for one of the Weekday's clients

Min Experience: 4 years

Location: Bengaluru

JobType: full-time

Requirements

Key Responsibilities

As a Data Engineer, you will play a crucial role in designing and maintaining scalable and high-performance data systems. Your responsibilities will include:

1. Data Pipeline Development and Management

• Design, build, test, and maintain efficient data pipelines and data management systems.
• Develop and manage ETL (Extract, Transform, Load) and ELT (Extract, Load, Transform) processes to integrate data from diverse sources such as databases, APIs, and real-time streams.

2. Data Modeling and Architecture

• Design data models and implement schemas for data warehouses and data lakes to support analytics and business operations.
• Optimize data storage, access, and performance for scalability and maintainability.

3. Data Quality and Integrity

• Implement validation, cleansing, and monitoring to maintain data accuracy, consistency, and reliability.
• Define and enforce best practices and standards for data governance and quality.

4. Infrastructure Management

• Manage and monitor key data infrastructure components including databases, data lakes, and distributed computing environments.
• Apply data security protocols and ensure proper access controls are in place.

5. Automation and Optimization

• Automate data workflows and pipelines to improve reliability and performance.
• Continuously monitor and fine-tune systems for operational efficiency.

6. Collaboration and Support

• Partner with data scientists, analysts, software engineers, and business stakeholders to gather requirements and provide scalable data solutions.
• Document processes, workflows, and system designs; support cross-functional teams with technical guidance.

7. Technology Evaluation

• Stay current with emerging tools and technologies in the data engineering space.
• Evaluate and recommend new solutions to enhance data capabilities and performance.

Education and Experience

• Bachelor’s or Master’s degree in Computer Science, Engineering, Information Systems, Data Science, or a related field.
• 5 to 7 years of experience in data engineering, software development, or a similar domain.

Technical & Functional Competencies

Required Skills & Qualifications

Technical Proficiency

• Programming: Strong experience in Python and SQL.
• Databases: Proficient in relational (PostgreSQL, MySQL) and NoSQL (MongoDB, Cassandra) databases.
• Data Warehousing & Lakes: Hands-on experience with platforms like Snowflake, Redshift, BigQuery.
• ETL/ELT Tools: Proficiency with tools like Apache Airflow, AWS Glue, Azure Data Factory, Talend.
• Big Data: Working knowledge of Apache Spark or similar big data technologies.
• Cloud Platforms: Experience with AWS, Azure, or GCP for data engineering workflows.
• Data Modeling: Strong understanding of modeling techniques and best practices.
• API Integration: Ability to build and consume APIs for data integration.
• Version Control: Experience with Git or other version control systems.

Soft Skills

• Analytical mindset with a strong problem-solving approach.
• Excellent communication skills for both technical and non-technical audiences.
• Team player with a collaborative work ethic.
• Detail-oriented with a commitment to data quality.
• Adaptability to new technologies and changing project requirements.

Key Skills:
ETL, Data Modeling, Data Architecture, Cloud Data Platforms, Python, SQL, Big Data, Data Warehousing, API Integration