Data Engineer

• Analyze business procedures and requirements to recommend specific types of data that can be used to improve upon them.
• Establish work estimates based on business requirements through participation in project feasibility, scoping, and estimating activities. 
• Designing and implementing data storage solutions: This involves designing and implementing storage solutions on Azure, such as Azure Data Lake Storage, Azure Blob Storage, and Azure SQL Database.
• Building and maintaining data pipelines: Responsible for building and maintaining data pipelines to move data from various sources to data storage solutions. This includes building ETL (Extract, Transform, Load) pipelines using Azure Data Factory or Azure Databricks.
• Managing data workflows: Responsible for managing data workflows and ensuring that data is processed in a timely and accurate manner.
• Monitoring data performance: Responsible for monitoring data performance and ensuring that data pipelines and workflows are running smoothly.
• Ensuring data quality: Responsible for ensuring that data is accurate, consistent, and reliable.
• Troubleshooting data issues: Responsible for troubleshooting data issues and resolving them in a timely manner.
• Collaborating with other teams: Responsible for collaborating with other teams, such as data analysts and data scientists, to ensure that their data needs are met.
• Ensuring data security: Responsible for ensuring that data is stored and processed securely, following best practices for data security.
• Ability to gain a complete understanding of systems, data flows, integration points, and quickly assess the impacts of changes.

Other Knowledge/Skills/Abilities:

• Excellent communication/interpersonal skills, with experience coordinating ETL work with an onshore/offshore vendor team. Must be able to clearly communicate.
• Excellent problem-solving skills. Applies technical knowledge to determine solutions and solve complex problems.
• Proven ability to create and maintain documentation (ETL designs, technical specs, test cases, test results, project status, deployment plans, etc.)
• Experience working on projects using Agile, Scrum, or Scaled Agile Framework (SAFe).

• Strong understanding of Spark architecture and concepts, as well as hands-on experience working with Spark RDDs, DataFrames, and SQL.
• Proficient in either Python, PySpark or Scala, which are the primary languages used to write Spark applications in Azure Databricks.
• Prior experience of working with cloud computing platforms like Azure, AWS, or Google Cloud, and be familiar with the tools and services offered by these platforms, such as Azure Blob Storage, Azure Data Factory, and AWS S3.
• Proficiency with SQL to facilitate the data integration process through data profiling, understanding relationships in the data, and validating the integrity of the data. 
• Experience working with enterprise class RDBMS platforms like Oracle, SQL Server, and DB2.  Must have worked on ETL efforts in one of these environments.
• Experience creating UNIX shell scripts to perform file level validation and analysis. 
• Familiarity with NoSQL databases and JSON formats, preferably CosmosDB