Data Engineer - Databricks (Performance & Security)
Tasks
- Collaborate on high availability logging and monitoring
- Contribute to Databricks Asset Bundles for deployment
- Contribute to Delta Live Tables feature enablement
- Fine tune PySpark jobs
- Implement and maintain governance, security, and permissions
- Implement performance and security benchmarks for production workloads
- Implement streaming with Structured Streaming
- Integrate Kafka for streaming
- Manage Databricks environments integrated with Airflow
- Monitor cluster and job performance
- Optimize Databricks performance across jobs clusters and workloads
- Troubleshoot and fine tune SQL queries
- Use Autoloader for data ingestion
- Use Terraform for infrastructure as code
- Use Unity Catalog for data governance
Perks/Benefits
Skills/Tech-stack
Active Directory | Airflow | Asset bundles | Autoloader | Azure Active Directory | Azure Data | Azure Data Lake | Azure Data Lake Storage | Azure Data Lake Storage Gen2 | Azure Kubernetes | Azure Kubernetes Service | Data Governance | Data Lake Storage Gen2 | Data Quality | Databricks | Databricks (asset) | Databricks Asset Bundles | Delta Live Tables | Delta Live) | GitLab | Grafana | Kafka | Kubernetes Service | PySpark | Python | SQL | Structured Streaming | Talend | Terraform | Unity Catalog
Education
N/A
Roles
Related jobs
-
AWS | Apache Flink | Apache Kafka | Apache Spark | Cloud platformAutonomy | Flexible time off | Flexible work hours | Fully remote | Global health coverageSenior-level Full TimePortugal R12h ago
-
Django | Docker | FastAPI | Flask | GitAgile work environment | Career development | Promotion from withinMid-level Full TimePorto17h ago
-
Data Engineer EUR 46K-65KAPI REST | AWS Athena | AWS EC2 | AWS RDS | Amazon ElastiCacheFlexible working hours | Professional development workshops | Work on sustainability projectsSenior-level Full TimeRua Santos Dias No. 1121, Senhora …17h ago
-
Machine Learning Engineer EUR 57K-85KAWS | Argo Workflows | ArgoCD | Azure | DatabricksAnnual learning budget | Equity | Home office starter budget | Hybrid work model | Private medical insuranceMid-level Full TimeLisbon, Portugal1d ago
-
(Senior) Software Engineer (AI / ML) EUR 56K-79KArtificial Intelligence | CI/CD | Computer Vision | Docker | Docker ComposeAnnual retreats | Company computer and monitor | Flexible working arrangements | Health insurance | Paid annual leaveSenior-level Full TimeLisboa, Lisboa, Portugal1d ago
-
Data Engineer Lead | BPCE-SI EUR 40K-55KData Engineering | Data Quality | DataStage | ETL | IBM DataStageElectric bikes | Green transportation budget | Hybrid work | Lactation Room | Prayer roomSenior-level Full TimeLisbon, Lisbon, Portugal1d ago
-
Data Engineer EUR 65K-65KAWS | ELT | ETL | GCP | Power BIEnglish communication support | Hybrid workMid-level Full TimeEspinho, Portugal1d ago
-
Machine Learning Engineer: Recommendations (Remote) USD 80K-120KA/B | A/B Testing | Apache Hive | Apache Spark | B testingApple laptops provided | Maternity leave | Paternity leave | Remote work | Stock optionsSenior-level Full TimePortugal - Remote R1d ago
-
Data Engineer (Qlik & Snowflake) EUR 28K-35KCloud Computing | Data Marts | Data Modeling | ETL | Qlik SenseCareer development plan | Diversity and inclusion culture | Flexible work schedule | Team support | Training opportunitiesEntry-level Full TimeLisboa, Portugal2d ago
-
Data Engineer EUR 61K-63KAWS | Apache Airflow | Apache Kafka | Apache Spark | Automated testingFlexible working hours | Hybrid work model | Lunch vouchers | Private health insurance | Transport subsidyMid-level Full TimePortugal, Porto, Professionals2d ago
-
Backend Engineer (Senior) ID55642 EUR 54K-76KAWS | Apache Spark | Application Profiling | C# | C++Education budget | Fitness budget | Flexible schedule | Growth roadmap | MentorshipSenior-level Full TimePorto Velho, Brazil3d ago
-
Senior Software Engineer – C# ID55642 EUR 57K-76KAWS | Apache Spark | Application Profiling | C# | C++Education budget | Fitness budget | Flexible schedule | Mentorship | Office optionsSenior-level Full TimePorto Velho, Brazil3d ago
-
Data Engineer (FinOps) ID56348 EUR 55K-65KAWS | Apache Airflow | CI/CD | DBT | DagsterEducation budget | Fitness budget | Flextime | Mentorship | Office optionsMid-level Full TimePorto Velho, Brazil3d ago
-
FinOps Data Engineer ID56348 EUR 55K-65KAWS | Apache Airflow | CI/CD | DBT | DagsterEducation budget | Fitness budget | Flexible schedule | Mentorship | Office optionsMid-level Full TimePorto Velho, Brazil3d ago
-
Cloud Data Engineer – FinOps ID56348 EUR 55K-65KAWS | Apache Airflow | CI/CD | DBT | DagsterEducation budget | Fitness budget | Flexible schedule | Mentorship | Office optionsMid-level Full TimePorto Velho, Brazil3d ago
-
Senior-level Full TimePortugal, Lisbon, Portugal4d ago
-
Senior Data Engineer – AI & Agentic Pipelines EUR 38K-63KAWS | Agile | Amazon Redshift | Apache Airflow | CI/CDFlexible working hours | Hybrid work model | Private health insurance | Training and development | Volunteer daySenior-level Full TimeLisbon, Portugal4d ago
-
Data Engineer EUR 36K-48K.NET | Apache Spark | Autoloader | C# | CI/CDHealth insurance | Hybrid work model | Remote work whenever possible | Work equipment providedMid-level Full TimeLisbon, Portugal4d ago
-
Data Engineer - AWS Mid/Senior EUR 54K-72KAWS | AWS CloudFormation | AWS Glue | AWS Lake Formation | AWS LambdaCareer growth programs | Health insurance | Hybrid work | Life insurance | Referral programSenior-level Full TimeLisboa, PT4d ago
-
Access Control | Airflow | Amazon Redshift | Amazon SageMaker | BigQueryFlexible work hours | Remote workSenior-level Full TimePortugal - Remote R4d ago
-
Senior Embedded Developer EUR 50K-68KAWS | Adaptive AUTOSAR | Agile | C++ | CI/CDEmployee health care | Flexible work hours | Hybrid work model | Life insurance | Professional career developmentSenior-level Full TimeLisboa, PT, 1200-246 R4d ago
-
Senior AI Developer Experience Engineer EUR 60K-85KAPI | Agentic Workflows | Azure DevOps | CI/CD | Claude CodeSenior-level Full TimePorto - Portugal4d ago
-
AI Software Engineer EUR 55K-79KCI/CD | Data Pipelines | Django | Docker | FastAPIAgile work environment | Career development | Promotion from withinMid-level Full TimePorto4d ago
-
Amazon Web Services | Apache Spark | Deep learning | Google Cloud | HadoopSenior-level Full TimeLisbon, Portugal5d ago
-
Senior Data Engineer EUR 55K-69KAirflow | Amazon S3 | ClickHouse | Data Lake | Data WarehouseCo-working | Company paid sick leave | English learning courses | Gym or swimming pool | Home office setup assistanceSenior-level Full TimePortugal - Remote R5d ago