Solution Architect- Data Engineering (2361)

CES has 26+ years of experience in delivering Software Product Development, Quality Engineering, and Digital Transformation Consulting Services to Global SMEs & Large Enterprises. CES has been delivering services to some of the leading Fortune 500 Companies including Automotive, AgTech, Bio Science, EdTech, FinTech, Manufacturing, Online Retailers, and Investment Banks. These are long-term relationships of more than 10 years and are nurtured by not only our commitment to timely delivery of quality services but also due to our investments and innovations in their technology roadmap. As an organization, we are in an exponential growth phase with a consistent focus on continuous improvement, process-oriented culture, and a true partnership mindset with our customers. We are looking for the right qualified and committed individuals to play an exceptional role as well as to support our accelerated growth.You can learn more about us at: http://www.cesltd.com/

Job Description

• Experience with Azure Synapse Analytics: Hands-on experience in designing, developing, and deploying solutions using Azure Synapse Analytics, including familiarity with its various components such as SQL pools, Spark pools, and Integration Runtimes.

• Expertise in Azure Data Lake Storage: In-depth understanding of Azure Data Lake Storage, including its architecture, features, and best practices for managing a large-scale Data Lake or Lakehouse in an Azure environment.

• Experience with AI Tools: Experience with AI Tools and LLMs (e.g. GitHub Copilot, Copilot, ChatGPT) in automating many of the responsibilities outlined for this role.

•  Knowledge of Avro and Parquet: Experience working with Avro and Parquet file formats, including data serialization, compression techniques, and schema evolution. Understanding of their advantages and use cases in a big data environment.

• Healthcare: Prior experience working with data in a healthcare or clinical laboratory environment and a strong understanding of PHI, GDPR & HIPPA/HITRUST is highly desirable.

• Certifications: Relevant certifications such as Azure Data Engineer Associate or Azure Synapse Analytics Developer Associate are highly desirable.

Essential Functions

• Data Integration and ELT Development: Design, develop, and maintain data pipelines for ingestion, transformation, and loading of data into Azure Synapse Analytics. This includes understanding functional and non-functional requirements, performing source data analysis, data profiling, and implementing efficient ELT processes.

• Azure Synapse Development: Work with Azure Synapse Analytics to build and optimize data models, SQL queries, stored procedures, and other artifacts necessary for data processing and analysis.

• Data Lake File Handling: Understand the characteristics of various file formats, optimizing data storage, and implementing efficient data reading and writing mechanisms for incremental updates within Azure Synapse Analytics.

• Data Governance and Security: Ensure compliance with data governance policies and implement security measures to protect sensitive data stored in Azure. This involves encryption, masking, and access control mechanisms.

• Performance Optimization: Continuously optimize data pipelines and storage configurations to improve performance, scalability, and reliability. This includes identifying bottlenecks, query tuning, and leveraging Azure Synapse Analytics features for parallel processing.

• Monitoring and Troubleshooting: Implement monitoring solutions to track data pipeline performance, data quality, and system health. Troubleshoot issues related to data ingestion, transformation, or storage, and provide timely resolutions.

Skills Needed to be Successful

•  Relational Database Experience: Proficiency with one or more of the following database platforms; e.g. Oracle, Microsoft SQL Server, PostgreSQL, MySQL/MariaDB

• Proficiency in SQL: Strong SQL skills, including experience with complex SQL queries, stored procedures, and performance optimization techniques. Familiarity with T-SQL for Azure Synapse Analytics is a plus.

• ELT and Data Integration Skills: Proven experience in building ELT pipelines and data integration solutions using tools like Azure Data Factory, Oracle Golden Gate, or similar platforms.

•  Ability to handle a variety of legacy data sources and file formats efficiently.

• Data Modeling and Warehousing Concepts: Familiarity with dimensional modeling, star schemas, and data warehousing principles. Experience in designing and implementing data models for analytical workloads.

• Analytical and Problem-Solving Abilities: Strong analytical skills with the ability to understand complex data requirements, troubleshoot technical issues, and propose effective solutions to meet business needs.

• Communication and Collaboration: Excellent communication skills with the ability to collaborate effectively with cross-functional teams, including Data Scientists, Reporting Analysts, and DevOps professionals.