Senior Data Engineer

We are seeking an experienced Senior Data Engineer to assist with the design, implementation, and optimization of components of our enterprise data architecture. In this role, you will architect scalable enterprise ETL/ELT solutions in Databricks, and drive technical best practices across our data engineering team. You'll be responsible for making critical technical and tactical decisions that shape our data ETL infrastructure while providing guidance to junior team members and collaborating with cross-functional stakeholders.

Key Responsibilities

• Own and architect production-grade data models, pipelines and infrastructure, making key technical decisions that ensure scalability, reliability, and performance
• Assist in design and implementation of complex data models to connect and integrate diverse data sources throughout the enterprise, including ERP system data, application data, and external data sources
• Lead technical initiatives to assist in modernizing our data platform, including evaluation and adoption of new technologies and tools
• Establish engineering standards for data quality, monitoring, and observability across data systems
• Develop architectural blueprints and data models for new data products and services that meet business requirements
• Mentor junior engineers on technical best practices and provide guidance on complex data engineering challenges
• Drive innovation in our ETL/ELT processes, implementing advanced automation and optimization techniques
• Present architectural decisions to technical and non-technical stakeholders, clearly articulating tradeoffs and benefits

Required Skills & Qualifications

• Bachelor's Degree in a STEM or computer-related field; other degrees acceptable with demonstration of technical skill
• 3+ years of professional experience in data engineering or similar roles
• Deep expertise in designing and implementing large-scale data processing systems
• Extensive experience with Python programming and data manipulation libraries, specifically Pandas and Pyspark
• Advanced SQL skills and experience in query optimization
• Proven track record of owning and delivering end-to-end data engineering projects
• Strong understanding of software engineering principles as applied to data systems Exceptional communication skills with ability to translate complex technical concepts to more junior coworkers as well as stakeholders
• Experience in the role of tech lead or project lead for production data pipeline development

Preferred Skills & Qualifications

• Experience architecting real-time data processing systems, including implementing CDC or streaming enabled pipelines
• Knowledge of data mesh or data fabric architectural patterns Experience with modern data lakehouse architectures, e.g. "medallion architecture"
• Expertise in optimizing Spark performance for large-scale data processing
• Experience implementing data governance frameworks and practices
• Experience with the Databricks ecosystem or an equivalent data platform
• Expertise in data pipeline orchestration tools (Airflow, Azure Data Factory, etc.)
• Experience with Infrastructure as Code and CI/CD for data platform