Data Engineer

Job Overview:

We are seeking a talented and motivated Data Engineer with a minimum 3 years of experience to join our dynamic team. The ideal candidate will possess a strong background in designing, building, and maintaining scalable data pipelines and databases. You will work closely with engineering teams and product owner to translate requirements into resilient, secure, and high-performance software. Key responsibilities include designing, coding, testing, and implementing new or existing software products based on specifications. A crucial aspect of this role is a commitment to continuous learning and adaptability to evolving technologies and market trends.

Key Responsibilities:

• Data Pipeline Development:
  • Design, build, and maintain robust and scalable data pipelines to process and transform large volumes of data from various sources.
  • Implement ETL (Extract, Transform, Load) processes to ensure data is accurately and efficiently integrated into data warehouses or data lakes.

Database Management:

• Develop, build and optimize database queries and schema designs to enhance performance and support analytical needs.

Data Integration:

• Collaborate with cross-functional teams to gather data requirements and integrate data from diverse sources including APIs, flat files, and databases.
• Ensure data consistency, accuracy, and integrity across various systems.Data Quality and Monitoring:
• Implement data quality checks and monitoring processes to identify and resolve data issues proactively.
• Develop and maintain data documentation and metadata for transparency and reproducibility.

Performance Tuning:

• Monitor and tune the performance of data pipelines and databases to ensure optimal operation and scalability.
• Troubleshoot and resolve data-related issues in a timely manner.

Collaboration and Communication:

• Work closely with various stakeholders to understand their data needs and provide appropriate solutions.
• Communicate technical concepts and solutions effectively to both technical and non-technical team members.

Continuous Learning and Adaptation:

• Stay current with emerging technologies, trends, and best practices in data engineering.
• Demonstrate a proactive attitude towards learning new tools and technologies, and adapt quickly to changes in the market and industry requirements.

Qualifications:

• Education:
  • Bachelor’s degree in Computer Science, IT, Engineering or a related field

Experience:

• 3+ hands-on experience in data engineering
• Proficiency with data warehousing solutions and ETL tools.

Technical Skills:

• Proficient in SQL: Extensive experience in writing complex queries, stored procedures, and functions in SQL and Azure SQL Managed Instance (SQL-MI).

• Data Pipeline Orchestration: Hands-on experience with data pipeline orchestration tools such as SSIS, Azure Data Factory, and Azure Synapse.
• Knowledge of Microsoft Fabric: Familiarity with end-to-end analytics and data platform Microsoft Fabric
• Programming Skills: Proficiency in Python and PySpark for data manipulation and analysis.
• Azure DevOps: Experience with Azure DevOps for version control, continuous integration, and deployment.
• Repository Management: Strong practical experience with code repositories such as Azure Repos and Bitbucket.
• Database Concepts: Solid understanding of Relational Database Management Systems (RDBMS) and data warehousing concepts.
• Cloud Platforms: Experience working with Azure – Azure Virtual Machines, Azure Storage Account, Azure Key Vault

Nice to have:

• Data Visualization: Knowledge / familiarity with data visualization tools like Power BI to create insightful and interactive reports.
• Programming Knowledge: Familiarity with additional programming languages such as

.NET, Java, or front-end technologies like React/JavaScript is a plus.

• Project Management & Issue Tracking: Familiarity/exposure in using Jira - creating/managing stories, sub-tasks, tasks, bugs

Soft Skills:

• Excellent problem-solving skills and attention to detail.
• Strong analytical and organizational abilities.
• Ability to work effectively both independently and as part of a team.
• Effective communication skills to interact with various stakeholders.
• Willingness to learn new technologies and adapt to rapid changes in the data engineering landscape.