Lead Databricks Engineer
Adelaide, SA, Australia
AUD 70K-80K (estimate) Senior-level Full Time
Tasks
- Build ML ready datasets
- Build batch and real-time pipelines
- Design Lakehouse platforms
- Develop and deploy Delta Live Tables
- Develop structured and unstructured ingestion frameworks
- Enable data science workflows
- Ensure security, privacy, compliance
- Implement CI/CD and DevOps best practices
- Implement Databricks data pipelines
- Implement data governance quality checks monitoring observability
- Integrate cloud data services
- Lead and mentor data engineering teams
- Manage infrastructure with Infrastructure as Code
- Manage metadata and lineage with Unity Catalog
- Optimize performance, cost, and reliability
- Own end-to-end project delivery
- Support real time analytics with Structured Streaming
- Translate business needs into data solutions
Perks/Benefits
- Career Development Programs
- Discounted health insurance
- Discounts on products and services
- Paid parental leave
- Training programs
- Well-being initiatives
Skills/Tech-stack
AWS Glue | Airflow | Amazon Redshift | Amazon S3 | Apache Spark | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure DevOps | Azure Synapse | Azure Synapse Analytics | CI/CD | Data Factory | Data Governance | Data Lake Storage | Data Lineage | Data Modelling | Data Quality | Data Warehousing | Databricks | Databricks Workflows | Delta Lake | Delta Live Tables | Delta Live) | GitHub | Infrastructure as Code | Metadata Management | Monitoring | Observability | PySpark | Python | SQL | Snowflake | Structured Streaming | Synapse Analytics | Terraform | Unity Catalog | “as-code”
Education
N/A
Roles
Regions
Countries
States
Related jobs
-
AI Solutions & Delivery Specialist AUD 72K-95KAWS | Agentic AI | Artificial Intelligence | Azure | Bias AssessmentEmployee network groups | Qantas Group staff travel discountMid-level Full TimeMelbourne, Victoria, Australia5h ago
-
Senior Backend Engineer - Experiments Team AUD 120K-140KA/B | A/B Testing | B testing | Data Modeling | Data WarehousingEquity packages | Flexible leave options | Flexible remote work | Inclusive parental leave | Wellbeing allowanceSenior-level Full TimeSydney, Australia R10h ago
-
Data Engineer AUD 105K-125KAccess Control | Anomaly Detection | Apache Spark | Audit Logging | Data ContractsAdditional paid leave | Employee discounts | Flexible working arrangements | Free counseling services | Gym accessMid-level Full TimeSydney, Australia R11h ago
-
Data & AI Engineer II USD 133K-166KAPIs | Azure | CI/CD | Dash | Databricks401k match | Employee stock purchase plan | Gym reimbursement | Health and welfare benefits | Paid leave for living donor programMid-level Full TimeBrisbane, CA2d ago
-
Azure AI Developer Consultant AUD 130K-165K.NET | API Management | App Service | Azure API | Azure API ManagementFlexible work hours | On-site customer visits | Work from homeMid-level Full TimeSydney, NSW, AU2d ago
-
Senior Databricks Consulting Engineer AUD 170K-200KAWS | Apache Spark | Cloud Architecture | Cloud platform | Data GovernanceContinuous learning and certification support | Remote friendly working environmentSenior-level Full TimeSydney, New South Wales, Australia3d ago
-
Bash | Cloud platform | Data Processing | Data pipeline | DockerAsynchronous culture | Entrepreneurial team | Friendly atmosphere | Hands-off managementMid-level Full TimeBrisbane, Australia3d ago
-
Senior Analytics Engineer AUD 140K-170KAutomation | BigQuery | Business Intelligence | DBT | Data GovernanceSenior-level Full TimeAU - HQ - NSW3d ago
-
Chief Platform Engineer, Data Platforms AUD 66K-200KArchitecture | Artificial Intelligence | Automation | Cloud Computing | Cost OptimizationEmployee discounts | Flexible work arrangements | Learning and development opportunities | Leave options | Volunteering opportunitiesExecutive-level Full TimeSydney, NSW, Australia3d ago
-
Senior Embedded Software Engineer AUD 140K-180KAMD Xilinx | AMD Xilinx Zynq | AMD Xilinx Zynq UltraScale plus MPSoC | ARM Cortex | ARM Cortex-A53Senior-level Full TimeSydney, Australia3d ago
-
Mid-level Full TimeSydney, Australia; Newcastle, Australia3d ago
-
AI Solutions & Delivery Lead AUD 121K-136KAWS | Agentic AI | Azure | Bias Assessment | Cloud platformEmployee network groups | Supportive inclusive culture | Travel discountSenior-level Full TimeMelbourne, Victoria, Australia3d ago
-
Customer Analytics & Insights Lead - Hawthorn East AUD 118K-127KA/B | A/B Testing | AI tools | B testing | Cohort AnalysisDiscounts | Fitness classes | Flexible working options | Free parking | Gym facilitySenior-level Full TimeHawthorn East Melbourne, VIC, AU, 3123 R3d ago
-
Senior Data Engineer AUD 160K-180KAI functions | AWS | Apache Airflow | Azure | Cortex AICareer growth opportunities | Flexible working hours | Health and Wellbeing Programmes | Hybrid work model | Learning and development programsSenior-level Full TimePerth, Australia3d ago
-
Access Control | Anomaly Detection | CI/CD | Classification | Deep learningCompetitive salary | Exposure to LLM RAG MLOps LLMOps Kubernetes | Hands on AI ML GenAI projects | Remote working model | Scalable secure governed observable engineering environmentSenior-level Full TimeAustralia3d ago
-
AI Security | API Development | Agent systems | Agentic AI | Backend DevelopmentContract engagement | Fully remote work model | Remote ownership and technical leadershipSenior-level Full TimeAustralia R4d ago
-
Lead Data Scientist - AI Technical Solutions AUD 142K-170KAI Pipelines | AWS | Apache Airflow | Artificial Intelligence | AzureContinuous learning | Flexible work arrangements | Remote WorkingSenior-level Full TimeSydney, Melbourne R4d ago
-
Mid-level Full TimeSydney, Australia4d ago
-
Senior AI Engineer (AI Innovation Office) AUD 143K-154KAPI | Agents | CI/CD | Cloud services | ComplianceEmployee discounts | Flexible work arrangements | Learning and development opportunities | Leave options | Volunteering opportunitiesSenior-level Full TimeSydney, NSW, Australia4d ago
-
AI Engineer x 2 (AI Innovation Office) AUD 120K-143KAPI Development | Artificial Intelligence | Automated testing | CI/CD | Cloud ComputingEmployee discounts | Flexible work arrangements | Learning and development | Leave options | Mortgage discountsMid-level Full TimeSydney, NSW, Australia4d ago
-
Lead AI Solutions Delivery Engineer AUD 160K-200KAPI Integration | Autonomous Agents | C++ | Claude | Cloud ComputingSenior-level Full TimeSydney, NS, Australia R5d ago
-
Senior Data Engineer AUD 146K-155KAnomaly Detection | CI/CD | DBT | Data Modeling | Data pipelineCareer growth opportunities | Employee discounts | Flexible working arrangements | Health and wellbeing servicesSenior-level Full TimeMelbourne, AU5d ago
-
Databricks Engineer AUD 100K-140KAPIs | AWS | Azure | CI/CD | Data ModelingCareer breaks | Career development program | Family Inclusive Workplace certification | Flexible work arrangements | Local and international career opportunitiesMid-level Full TimeSydney, International House, 3 Sussex St, …5d ago
-
Senior Data Engineer AUD 129K-138KApache Spark | Artificial Intelligence | Azure Data | Azure Data Factory | Azure DevOpsEmployee assistance program | Fitness passport | Flexible work options | Meals and Entertainment Salary Packaging | Professional development and learning opportunitiesSenior-level Full TimeAnglicare Hub Macquarie Park, Australia5d ago
-
Senior AI Engineer AUD 141K-152KAPI Design | Agentic AI | Azure | Bicep | CI/CDCorporate partner discounts | Employee discounts | Flexible work environment | Hybrid working | RACV club membership discountsSenior-level Full Time485 Bourke Street Melbourne5d ago