Senior Data Engineer - Databricks
Tasks
- Apply CI CD best practices for pipeline development
- Apply role based access control and secrets management
- Build automation to reduce manual intervention
- Build data quality monitoring and observability
- Design and build Databricks pipelines to ingest transform and serve ERP and CRM data
- Enforce data quality and incremental processing patterns
- Enforce multi tenant data isolation patterns
- Implement Delta Lake architecture including schema design and partitioning
- Implement data security and compliance best practices
- Integrate data pipelines with AI and analytics ecosystem
- Maintain and report SLA performance metrics
- Maintain technical documentation runbooks and architecture decision records
- Migrate legacy ETL and ELT pipelines to Databricks
- Monitor, alert, and respond to incidents
- Optimize data pipelines to reduce compute costs and processing windows
- Own Databricks production support
- Perform on call incident response across time zones
- Provision validate and harden tenant data pipelines
Perks/Benefits
Skills/Tech-stack
ACID transactions | AWS | Access Control | Apache Spark | Automated testing | Azure | Azure Databricks | CI/CD | Data Governance | Data Observability | Data Partitioning | Data Quality | Data Security | Databricks | Delta Lake | Deployment tooling | ELT | ETL | Incremental processing | Multi-tenant | Multi-tenant architecture | PostgreSQL | PySpark | Python | Role-Based Access | Role-Based Access Control | SQL | Schema evolution | Secrets management | Streaming | Tenant architecture | Version control
Education
N/A
Roles
Related jobs
-
Senior Software Engineer - Testing AI Coding Agents USD 100K-100KDocker | FastAPI | Functional testing | Integration Testing | JavaScriptFlexible schedule | Project based workSenior-level Full TimePoland - Remote R13h ago
-
Senior Software Engineer - AI Model Evaluation USD 80K-80KCase Development | Docker | FastAPI | Feedback iteration | Functional testingSenior-level Full TimePoland - Remote R13h ago
-
Senior Software Engineer - AI Coding Benchmarks USD 80K-80KDocker | FastAPI | Functional testing | Integration Testing | JavaScriptSenior-level Full TimePoland - Remote R13h ago
-
Product Intelligence Engineer PLN 258K-329KAutogen | CrewAI | Data Governance | Data Modeling | Function CallingCareer development | Distributed team collaboration | Learning opportunities | Work from anywhereSenior-level Full TimeBielsko-Biała, Poland R22h ago
-
Embedded Software Engineer (Poland) PLN 180K-338KAnalog circuits | Automated testing | BLE | Bash | Bluetooth Low EnergyEducation Training Programs | English teachers | Medical insurance | Paid time off | Remote workSenior-level Full TimeKraków, Poland R1d ago
-
Asset Management | Data Analysis | Failure Mechanisms | Predictive Modeling | Probabilistic ModelingAnnual performance bonus | Disability programs | Flexible work hours | Hybrid work | Life insuranceSenior-level Full TimePL-PDK-WARSAW-ALEJA KRAKOWSKA 110/114 BLD 3014613, Poland R1d ago
-
Test - nazwa stanowiska PLN 237K-328KAI Agents | APIs | Amazon Web Services | Automated testing | AzureBuddy onboarding program | Certificate training programs | Flexibility in working hours | Remote work option | Udemy learning platformSenior-level Freelance Full TimePoland R2d ago
-
Apache Airflow | BigQuery | Cloud platform | DBT | Data WarehousingAnnual off-site events | Co-working space allowance | Employee stock ownership program | Flexible asynchronous schedule | Fully remoteSenior-level Full TimePoland R2d ago
-
Senior AI Engineer (Remote) PLN 211K-310KAPIs | AWS | Agentic Systems | Azure | Backend DevelopmentHybrid work | Professional development | Remote work | Stock options program | Wellbeing supportSenior-level Full TimeWarsaw R2d ago
-
Data Engineer PLN 140K-224KApache Airflow | Apache Spark | CI/CD | DBT | Data LakehouseAccess to e-learning platform | Hybrid work | Life insurance | Lunch subsidy | Medical insuranceMid-level Full TimeŁódź, PL, 90-118 R3d ago
-
Senior Data Engineer PLN 175K-267KApache Airflow | Apache Spark | CI/CD | Cloud platform | DBTE-learning platform | Hybrid work | Life insurance | Lunch subsidy | Medical insuranceSenior-level Full TimeŁódź, PL, 90-118 R3d ago
-
Expert Data Engineer PLN 208K-320KApache Airflow | Apache Spark | CI/CD | Cloud platform | DBTE-learning platform | Hybrid work | Life insurance | Lunch subsidy | Medical insuranceSenior-level Full TimeŁódź, PL, 90-118 R3d ago
-
Alerting | BigQuery | CI/CD | Cloud Run | Cloud SQLContinuous learning | Flexible working arrangements | Fully remote | High autonomy | International engineering teamSenior-level Full TimePoland R3d ago
-
Associate AI Engineer PLN 139K-139KAI Evaluation | API Development | AWS | Agent systems | AzureEmployer supported pension plan | Extra paid holidays | Flexible benefits | Home-office allowance | Hybrid work modelMid-level Full TimeWarszawa, Mazowieckie, PL R5d ago
-
Data & Advanced Process Analytics Engineer PLN 276K-276KArtificial Intelligence | Data Visualization | ETL | Machine Learning | Process miningHybrid workMid-level Full TimePoland - Warsaw R6d ago
-
Senior-level Contract Full Timewarsaw, Poland - Remote R6d ago
-
Data Engineer | NDA PLN 276K-276KApache Airflow | Apache Spark | Azure Data | Azure Data Lake | CI/CDMid-level Full TimePoland - Remote R7d ago
-
Software Developer / Data Engineer PLN 276K-276KAPI Design | AWS | Airflow | Angular | AzureHybrid work modelMid-level Full TimePoland - Warsaw R7d ago
-
DevOps Engineer (AI Inference) USD 184K-287KAnsible | Argo CD | Bash | CI/CD | Cluster APIExtra Paid Sick Leave | Extra paid vacation | Flexible working hours | Hybrid work | Language coursesMid-level Full TimePoland, Serbia,, Cyprus, Georgia R8d ago
-
DevOps Engineer (AI Inference) USD 152K-287KAnsible | Argo CD | Bash | CI/CD | CNIExtra Paid Sick Leave | Extra paid vacation | Flexible working hours | Hybrid or remote options | Language coursesMid-level Full TimePoland, Serbia,, Cyprus, Georgia R8d ago
-
Senior Data Engineer (Integrations / Data Platform) PLN 220K-384KData Modeling | Data Quality | Event Driven | Event-driven architecture | GCP DataflowRemote workSenior-level Full TimeWarsaw, Masovian Voivodeship, Poland R9d ago
-
Senior Data Engineer (Integrations / Data Platform) PLN 220K-384KCloud platform | Data Modeling | Event Driven | Event-driven architecture | GCP DataflowRemote workSenior-level Full TimeWarsaw, Masovian Voivodeship, Poland R9d ago
-
Senior AI Engineer PLN 246K-352KAI Agents | AWS | Agent Orchestration | Apache Beam | Apache SparkFlexible location within Europe Ukraine and LATAM | Remote workSenior-level Full TimeWarsaw, Masovian Voivodeship, Poland R9d ago
-
AI Architecture Engineer - Supply Chain Team (f/m/d) PLN 206K-254KAmazon Web Services | Artificial Intelligence | Azure | Data Architecture | Data PipelinesContinuous improvement culture | On-site work | Remote work flexibilitySenior-level Full TimePOL - Remote, Poland R9d ago
-
Senior/Lead Software Data Engineer (Roads Team) USD 75K-112KAWS | AWS Glue | AWS IAM | Amazon Athena | Amazon EC2Flexible work schedule | Health care | Parental leave | Remote work optionSenior-level Full TimeMapbox Poland R10d ago