Lead Databricks Engineer
Adelaide, SA, Australia
AUD 70K-80K (estimate) Senior-level Full Time
Tasks
- Build batch and real-time data pipelines
- Design lakehouse platforms on Databricks
- Develop structured and unstructured data ingestion frameworks
- Drive DevOps best practices
- Enable ML ready datasets for data science workflows
- Ensure security and privacy compliance
- Establish coding best practices
- Implement CI/CD pipelines
- Implement data governance quality checks monitoring and observability
- Implement scalable data platforms with Spark and Delta Lake
- Integrate cloud services for data processing
- Lead and mentor data engineers
- Manage metadata and data lineage with Unity Catalog
- Optimize performance, cost, and reliability
- Own end to end project delivery and stakeholder translation
- Support real time analytics with Structured Streaming and Delta Live Tables
Perks/Benefits
- AI Driven Learning Platform
- Career development training programs
- Discounted health insurance
- Discounts on products and services
- Paid parental leave
- Recognition and rewards
- Wellbeing initiatives
Skills/Tech-stack
AWS | AWS Glue | Airflow | Amazon Redshift | Amazon S3 | Apache Spark | Azure | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure DevOps | Azure Synapse | Azure Synapse Analytics | Batch Processing | CI/CD | Data Factory | Data Governance | Data Lake Storage | Data Lineage | Data Modeling | Data Pipelines | Data Quality | Data Warehousing | Data orchestration | Databricks | Databricks Workflows | Delta Lake | Delta Live Tables | Delta Live) | DevOps | GCP | GitHub | Infrastructure as Code | Medallion Architecture | Metadata Management | Monitoring | Observability | PySpark | Python | Real Time | Real-time analytics | SQL | Security Compliance | Snowflake | Structured Streaming | Synapse Analytics | Terraform | Unity Catalog | “as-code”
Education
N/A
Regions
Countries
States
Related jobs
-
Automation | Big Data | CI/CD | Cloud platform | Data ArchitectureFlexible work from anywhere | International collaboration | Remote-first work environment | Work-life balanceExecutive-level Full TimeAustralia R21h ago
-
Data Analytics Manager AUD 113K-125KAutomation | Cloud Platforms | Data Engineering | Data Modeling | Data PipelinesCo contribution superannuation | Health and wellbeing program | Hybrid work arrangements | Learning and development | Study assistanceMid-level Full TimeBrisbane CBD1d ago
-
Engineer, Machine Learning Systems (AI Products) AUD 180K-200KAWS | Apache Airflow | Apache Spark | Dask | Data orchestrationAccess to modern office spaces | Collaborative boost days | Hybrid work model | Mentoring and coaching cultureSenior-level Full TimeAU: Sydney (45 Clarence St)1d ago
-
Data Engineer AUD 89K-111KAirflow | Azure Data | Azure Data Factory | Code Repository | Code reviewCareer development | Flex leave | Flexible work arrangements | Paid parental leave | Wellness incentivesEntry-level Full TimeMelbourne, VIC, AU, 30001d ago
-
Databricks Platform Engineer AUD 90K-112KAWS | Access Control | Access Management | Apache Spark | AutomationCareer development | Comprehensive benefits package | Flex leave | Flexible work arrangements | Paid parental leaveSenior-level Full TimeMelbourne, VIC, AU, 30001d ago
-
Senior Generative AI Developer (GCP) AUD 130K-180KBigQuery | BigQuery ML | CI/CD | Chroma | Cloud platformFlexible working | Hybrid working | Income protection insurance | Leave options | Parental leaveSenior-level Full TimeMelbourne, Australia1d ago
-
Lead Gen AI Engineer (GCP) AUD 130K-146KAPI | CI/CD | Cloud platform | Continuous Testing | GCPFlexible working | Hybrid working | Income protection insurance | Parental leave | Study leaveSenior-level Full TimeMelbourne, Australia1d ago
-
AWS Glue | AWS Lambda | AWS Step Functions | Agile | Amazon AthenaHybrid work flexibilitySenior-level Contract Full Time TemporarySydney, NSW - CBP North, 1 …1d ago
-
Senior-level Full TimeAUS-Sydney-Blue-Street, Australia1d ago
-
Data Engineer AUD 94K-130KAWS CDK | AWS Glue | Apache Airflow | Apache Flink | Apache IcebergExtra annual leave days | Leadership development programs | LinkedIn Learning | Perkbox discountsMid-level Full TimePyrmont, New South Wales, Australia1d ago
-
Senior Quantitative Data Engineer - 2-year fixed-term AUD 122K-127KAWS | Agile methodology | Algorithmic trading | Apache Spark | AzureFlexible interview adjustments | Flexible working | Hybrid workingSenior-level Full Time TemporarySydney Office, Australia1d ago
-
Senior AI Engineer AUD 131K-162KAI Foundry | AI Services | Agile | Azure AI | Azure AI FoundryAccess to modern cloud technologies | Collaborative culture | Free onsite parking | Hybrid work | Work from home 2 days per weekSenior-level Full TimeMulgrave, VIC, 3170, AU1d ago
-
Test Lead - Data AUD 90K-125KAlteryx | Data Governance | Data Quality | Data Validation | Data profilingSenior-level Full TimeAustralia2d ago
-
Senior Engineer - ML Systems (AI Products) AUD 175K-200KAWS | Apache Airflow | Apache Spark | Dask | Data orchestrationHybrid work model | Mentorship | On call ownershipSenior-level Full TimeAU: Sydney (45 Clarence St)2d ago
-
Staff AI & Data Engineer AUD 150K-200KAgentic AI | Anthropic | Apache Kafka | Azure | Azure EventFlexible working environment | Health and safety support | Inclusive workplaceSenior-level Full TimeAirTrunk HQ - North Sydney2d ago
-
Senior Data Engineer — Platform & Architecture AUD 139K-146KCI/CD | Clustering | Cost Optimization | Data Governance | Data ModelingEAP support | Fitness passport | Salary packagingSenior-level Full TimeBrisbane, Queensland 4000, Australia2d ago
-
Data Platform Lead AUD 140K-155KAccess Control | Cloud Data | Cloud data platform | Compliance Management | Data ArchitectureFlexible working arrangements | Hybrid work | Innovation rewards | Operational support | Professional developmentSenior-level Full TimeAdelaide, South Australia 5000, Australia2d ago
-
Senior Data Engineer — Integration & Pipelines AUD 118K-120KAPI pagination | Anomaly Detection | CI/CD | Data Governance | Data QualityEAP support | Fitness passport | Salary packagingSenior-level Full TimeBowen Hills, Queensland 4006, Australia2d ago
-
Senior Machine Learning Engineer AUD 131K-148K3D Perception | Airflow | C++ | Computer Vision | Continuous MonitoringEquity grants | Health insurance | Recovery BenefitsSenior-level Full TimeSydney, New South Wales, Australia2d ago
-
Data Engineer (AWS Cloud) AUD 129K-142KAWS Aurora | AWS Aurora RDS | AWS CodeCommit | AWS EMR | AWS GlueFlexible start and finish times | Flexible work arrangements | Job share options | Part-time arrangements | Work location flexibilitySenior-level Full TimeSydney, NSW - CBP North, 1 …2d ago
-
Data Governance Lead AUD 125K-135KAccess Control | Cause analysis | DAMA-DMBOK | Data Classification | Data ComplianceSenior-level Full TimeHJ Head Office, Australia2d ago
-
Staff Software Engineer - Machine Learning AUD 121K-125KAWS | Airflow | BigQuery | CI/CD | Data DriftCommunity grants | Flexible leave | Hackdays | Hybrid work | Matched payroll givingSenior-level Full TimeRichmond, Melbourne VIC, Australia2d ago
-
Senior Embedded Software Engineer (Onsite) USD 86K-165KASIC architecture | Abstraction layer | BSP | Board Bring-up | Bring-up401k match | Dental insurance | Disability coverage | Employee assistance program | Health wellness incentivesSenior-level Full TimeUS-FL-MELBOURNE-313 ~ 1344 S Babcock St …2d ago
-
GCP Data Engineer - Data Modelling AUD 70K-160KApache Airflow | BigQuery | Cloud platform | Composer | Data ModelingCareer development | Learning opportunitiesMid-level Full TimeMELBOURNE, VIC, AU, 30082d ago
-
AI Solutions Architect / AI Engineer AUD 170K-180KAI Agents | API Development | Automated testing | CI/CD | Cloud NativeHybrid work | Paid time offSenior-level Full TimeSurry Hills, NSW, Australia2d ago