Sr. Databricks Azure Data Engineer

USD 139K-237K (estimate) Senior-level Contract Found 12d ago

Tasks

• Build automated Parity Test Suite for data validation
• Design data solutions on Spark and Flink
• Develop Python SDK for Jupyter Notebooks
• Document solutions and workflows
• Enforce data governance, security, and quality standards
• Implement data quality rules
• Maintain CI/CD pipelines
• Manage Azure and Databricks resources
• Monitor and optimize data pipelines
• Participate in agile activities
• Transform and prepare data using SQL, Python, Java

Perks/Benefits

• N/A

Skills/Tech-stack

Apache Spark | Apache Spark internals | Azure DevOps | Azure Security | Azure ecosystem | CI/CD | Data Governance | Data Modeling | Data Quality | Data Warehousing | Databricks | Dataframes | Deequ | Delta Lake | GitHub | Great Expectations | JUnit | PySpark | Pytest | Python | Spark internals