Senior Data Engineer

Job ref: QVRRR546

Responsibilities:

• Design, develop, test, deploy and monitor data pipelines in Databricks on AWS from a wide variety of data sources.
• Design, develop, test, deploy and monitor scalable code with PySpark and SQL in Databricks.
• Identify opportunities to improve internal process through code optimisation and automation.
• Build data quality dashboards, lineage flows / and or monitoring tools to utilize the data pipeline, providing active monitoring and actionable insight into overall data quality and data governance.
• Assist in migrating data from legacy systems onto newly developed solutions.
• Follow and lead best practices on all data security, retention, and privacy policies.

Requirements:

• 5+ years experience of building ETL/ELT pipelines.
• Proven competency in solution design, development, implementation, reporting and analysis.
• Proficiency in Apache-Spark, Python and SQL languages.
• Proficiency in working with Text, Delta, Parquet, JSON, CSV, and XML data formats.
• Working knowledge of Spark structured streaming.
• AWS infrastructure experience, specifically working with S3.
• Solid understanding of git-based version control, DevOps, and CI/CD. Experience of working on Atlassian stack a plus.
• Knowledge of common web API frameworks and web services.
• Strong teamwork, relationship, and client management skills, and the ability to influence peers and senior management to accomplish team goals.
• Willingness to embrace modern technology, best practice, and ways of work.