Senior Data Engineer - Databricks
Tasks
- Apply CI CD best practices for pipeline development
- Apply role based access control and secrets management
- Build automation to reduce manual intervention
- Build data quality monitoring and observability
- Design and build Databricks pipelines to ingest transform and serve ERP and CRM data
- Enforce data quality and incremental processing patterns
- Enforce multi tenant data isolation patterns
- Implement Delta Lake architecture including schema design and partitioning
- Implement data security and compliance best practices
- Integrate data pipelines with AI and analytics ecosystem
- Maintain and report SLA performance metrics
- Maintain technical documentation runbooks and architecture decision records
- Migrate legacy ETL and ELT pipelines to Databricks
- Monitor, alert, and respond to incidents
- Optimize data pipelines to reduce compute costs and processing windows
- Own Databricks production support
- Perform on call incident response across time zones
- Provision validate and harden tenant data pipelines
Perks/Benefits
Skills/Tech-stack
ACID transactions | AWS | Access Control | Apache Spark | Automated testing | Azure | Azure Databricks | CI/CD | Data Governance | Data Observability | Data Partitioning | Data Quality | Data Security | Databricks | Delta Lake | Deployment tooling | ELT | ETL | Incremental processing | Multi-tenant | Multi-tenant architecture | PostgreSQL | PySpark | Python | Role-Based Access | Role-Based Access Control | SQL | Schema evolution | Secrets management | Streaming | Tenant architecture | Version control
Education
N/A
Roles
Related jobs
-
[C3F] Data Platform Engineer PLN 258K-384KAKS | Access Control | Azure Bicep | Azure Data | Azure Data FactoryFlexible employment | Internal and external training | International business trips | Language classes | Multisport cardMid-level Full TimeWarsaw, Masovian Voivodeship, Poland R1d ago
-
Senior Gen AI Software Engineer PLN 237K-340KAWS Bedrock | AWS Lambda | AWS SageMaker | Agent systems | Agentic AIEmployee Capital Plans PPK | Hybrid work model | Life insurance | Medical care | Multisport cardSenior-level Full TimeKrakow, Malopolskie, Poland R1d ago
-
Data Engineer PLN 228K-320KApache Airflow | Apache Spark | CI/CD | Dagster | Data ModelingAccess to benefits platform | E-learning platform | Equipment provided | Extra day off | Group insuranceMid-level Full TimeWarsaw R1d ago
-
Senior-level Full TimePoland R1d ago
-
Automation | Azure Data | Azure Data Explorer | CSV | ComplianceCross-functional collaboration | Exposure to large scale security telemetry | Opportunity to contribute to threat detection incident response and compliance | Professional growth | Remote work opportunityMid-level Full TimePoland R1d ago
-
AI Engineer, Assistant Vice President PLN 182K-182KAWS | Agentic Workflows | Azure | Cloud Computing | Data PrivacyAdditional holidays | Hybrid work model | Language classes | Life insurance | Medical packageExecutive-level Full TimeKrakow, Poland R1d ago
-
AI Engineer, Assistant Vice President PLN 182K-182KAPI Development | AWS | Agentic AI | Azure | Cloud PlatformsAdditional holidays | Hybrid work model | Language classes | Life insurance | Medical packageExecutive-level Full TimeKrakow, Poland R1d ago
-
Senior AI Architect (w/m/x) PLN 213K-254KAPI Integration | Agent systems | Agentic Workflows | Amazon Web Services | AzureContinuous learning environment | Flexible work schedule | Global career opportunities | Hybrid work model | Mentoring supportSenior-level Full TimeWarsaw, Poland R1d ago
-
AI Lead Developer (w/m/x) PLN 230K-254KAPI Integration | AWS | Agile | Azure | Cloud ComputingContinuous learning environment | Diverse global career opportunities | Flexible work schedule | Hybrid work model | Mentoring supportSenior-level Full TimeWarsaw, Poland R1d ago
-
Senior Data Engineer (KDB) PLN 189K-291KAlerting | Bash | CI/CD | Data Engineering | Data GovernanceE-learning platform | Hybrid work | Life insurance | Lunch subsidy | Medical insuranceSenior-level Full TimeKraków, PL, 30-302 R1d ago
-
AI/ML | API Development | APIs | Cloud infrastructure | Data GovernanceE-learning access | Hybrid work | Life insurance | Lunch subsidy | Medical insuranceEntry-level Full TimeKraków, PL, 30-302 R1d ago
-
Senior Data Consultant (KDB) PLN 250K-320KAPIs | Bash | CI/CD | Data Governance | Disaster RecoveryHybrid workSenior-level Full TimeKraków, PL, 30-302 R1d ago
-
Senior Data Engineer PLN 216K-300KAWS | Agile | Algorithms | Aster Data | AzureCompany paid premium medical package | Company-paid certifications | Conferences Workshops | Full remote working possibilities | Internal eventsSenior-level Full TimePoland R2d ago
-
AWS | Azure | Azure Synapse | Bicep | CI/CD100 percent remote workSenior-level Full TimeSzczecin, Poland R2d ago
-
Senior Data Engineer (Healthcare domain) PLN 204K-336KAzure | Data Governance | Data Lake | Data Modeling | Data WarehousingRemote workSenior-level Full TimeWarsaw, Masovian Voivodeship, Poland R2d ago
-
Senior Data Engineer (Healthcare domain) PLN 204K-336KAzure | Data Governance | Data Lake | Data Modeling | Data WarehousingRemote workSenior-level Full TimeWarsaw, Masovian Voivodeship, Poland R2d ago
-
AI Engineer PLN 179K-268KAgentic Workflows | BM25 | GRPC | Grafana | Hybrid searchE-learning platform | English learning support | Equipment provided | Extra day off | Group insuranceSenior-level Full TimeWarsaw R2d ago
-
API Design | API Integration | AWS | Alerting | AzureFlexible working hours | Hackathons | Partial remote work | Team-building tripsMid-level Contract FreelancePoland R2d ago
-
Mid-level Full TimePL-Warsaw-Lixa C R2d ago
-
AI Engineer, Assistant Vice President PLN 182K-182KAPI Development | Agent workflow | Amazon Web Services | Azure | Cloud ComputingAdditional holidays | Hybrid work model | Language classes | Life insurance | Medical packageExecutive-level Full TimeKrakow, Poland R2d ago
-
API Management | AWS | Azure | GCP | Kong APIHybrid work | Permanent employmentSenior-level Full TimeGdynia, PL, 81-537 R2d ago
-
Generative AI Solutions Engineer PLN 206K-254KAI Foundry | Agentic Workflows | Azure | Azure AI | Azure AI FoundryFlexible hybrid working model | Individual development plans | Professional development plan | Training opportunitiesMid-level Full TimeWarszawa, Poland R2d ago
-
API Versioning | Async frameworks | Asynchronous programming | Batch Processing | CI/CDHybrid workMid-level Full TimeKrakow R2d ago
-
AI Engineer PLN 183K-285KAgent systems | Automation | Azure AI | Azure Governance | CI/CDDigital course library | Employee assistance program | Hybrid schedule | Mindfulness program | Paid volunteer time offMid-level Full TimePoland Katowice (Francuska 46) R2d ago
-
Staff Software Engineer (Agentic Engineering) GBP 113K-170KAgentic Workflows | Amazon Web Services | Context Management | Distributed Systems | DockerSenior-level Full TimeUnited Kingdom R3d ago