Principal Data Engineer (Apache Spark, dbt, Airflow)
Sydney Office, Australia
AUD 140K-146K (estimate) Senior-level Full Time
Tasks
- Design build and maintain reusable data engineering frameworks
- Design data pipelines with scalability fault tolerance and cost efficiency
- Design to run for operational reliability
- Drive DataOps adoption with CI/CD automated testing and observability
- Embed data privacy lineage tracking and regulatory controls
- Ensure data pipelines and processes meet engineering standards and controls
- Evolve end to end data platform design
- Lead data lake and data mesh architecture design
- Manage embedded platform data engineers deliveries and stakeholders
- Provide enterprise best practices for Spark dbt and Airflow
- Standardize and accelerate data pipeline development
Perks/Benefits
Skills/Tech-stack
AWS | Amazon Redshift | Apache Airflow | Apache Iceberg | Apache Spark | Automated testing | Avro | Batch Processing | CI/CD | Cost Optimization | DBT | Data Lake | Data Lakehouse | Data Lineage | Data Privacy | Data Warehouse | Data mesh | DataOps | Distributed Systems | Fault Tolerance | Infrastructure as Code | JSON | Observability | Parquet | Regulatory Compliance | Scalability | Stream processing | Terraform | XML | “as-code”
Education
N/A
Roles
Data Engineer | Engineer | Principal | Principal Data Engineer
Regions
Countries
States
Cities
Related jobs
-
Senior AI Engineer AUD 160K-180KAWS | Agent Orchestration | Agentic AI | Artificial Intelligence | AutogenCareer development | Collaborative team environment | Stable work environmentSenior-level Full TimeMelbourne, Victoria 3000, Australia14h ago
-
Senior AI Engineer AUD 160K-180KAI-generated code | AWS | Agent Orchestration | Agentic AI | Artificial IntelligenceCareer developmentSenior-level Full TimeSydney, New South Wales 2000, Australia14h ago
-
Apache Spark | Azure | CI/CD | Containerization | Data ModelingFlexible working options | Paid parental leave | Paid volunteering days | Wellbeing leaveMid-level Full TimeMelbourne, VIC, AU21h ago
-
Databricks Data Engineer Senior Analyst AUD 70K-120KAgile | Apache Spark | Azure Blob | Azure Blob Storage | Azure DataSenior-level Full TimeBrisbane, 10 Eagle Street, Australia21h ago
-
Solutions Engineer AUD 140K-180K.NET | Apache Flink | Apache Hadoop | Apache NiFi | Apache SparkCareer development | Employee resource groups | Flexible work from home | Paid time off | Paid volunteer timeSenior-level Full TimeAustralia-New South Wales-Sydney R21h ago
-
AI Agents | AI Foundry | AI Search | Agent Framework | AutogenSenior-level Full TimeSydney, International House, 3 Sussex St, …21h ago
-
Benchmarking | CI/CD | Document Intelligence | Embeddings | Explainable AIFully remote work | Healthcare coverage | International opportunities | Long-term contract | Technical ownershipSenior-level Full TimeAustralia R1d ago
-
AWS | Airflow | Azure | Azure Data | Azure Data FactoryCareer Development Programs | Discounted health insurance | Discounts on products and services | Mentorship | Paid parental leaveSenior-level Full TimeMelbourne, VIC, Australia1d ago
-
Data and Analytics Senior AUD 130K-135KApache Spark | Azure Data | Azure Data Lake | Azure Data Lake Storage | Data FactoryCareer development | Part-time telework | Time to rechargeSenior-level Full TimeBrisbane, QLD, AU1d ago
-
Software Engineer - AI Developer Tooling Workflows AUD 100K-100KAI Agents | APIs | CI/CD | Continuous Deployment | Continuous integrationEquity packages | Flexible leave options | Hybrid work flexibility | Inclusive parental leave | Wellbeing allowanceMid-level Full TimeSydney, Australia R1d ago
-
Senior Data Engineer — Integration & Pipelines AUD 118K-120KAPI Integration | Anomaly Detection | Apache Spark | CI/CD | Data GovernanceCareer progression | EAP support | Fitness passport | Salary packaging | Social eventsSenior-level Full TimeBowen Hills, Queensland 4006, Australia1d ago
-
Senior Data Engineer — Platform & Architecture AUD 140K-147KCI/CD | Cost Optimization | Data Clustering | Data Governance | Data Lifecycle ManagementCareer progression | EAP support | Fitness passport | Modern office facilities | Salary packagingSenior-level Full TimeBrisbane, Queensland 4000, Australia1d ago
-
Data Engineer AUD 115K-120KAzure Data | Azure Data Factory | Azure Synapse | Azure Synapse Analytics | BI semantic modelsCommunity service leave | Cultural leave | Discounts | Flexible working | Inclusive cultureMid-level Full TimeVIC - Melbourne - 655 Collins … R1d ago
-
Principal Data Platform Engineer (AWS) AUD 150K-170KAWS | AWS Glue | AWS Lambda | Amazon EMR | Amazon RedshiftFlexible work location | Flexible working hours | Job-sharing | Part-time arrangements | Technology hubs in Melbourne and PerthSenior-level Full TimeSydney, NSW - CBP North, 1 …1d ago
-
Principal Data Scientist AUD 133K-146KAWS | Agentic AI | Apache Spark | Azure | CI/CDFlexible work arrangementsSenior-level Full TimeEveleigh, NSW - 5-7 Central Ave, …1d ago
-
Senior AI Engineer - Data Scientist (Sydney/Melbourne) AUD 143K-180KAPIs | Agent systems | Autogen | CI/CD | Computer VisionCertification support | Global mobility | Inclusive culture | Learning platformsSenior-level Full TimeMelbourne, AU1d ago
-
Cloud / Infrastructure Architect — JAPAC Flex AUD 128K-180KBigQuery | CI/CD | Cloud Build | Cloud Composer | Cloud FunctionsSenior-level Full TimeAustralian Capital Territory, Australia2d ago
-
Staff Engineer Chapter Lead – Generative AI AUD 137K-152KAWS | AWS Cloud | AWS Cloud Development Kit | AWS cloud development | AutogenSenior-level Full TimeSydney, NSW - CBP South, 11 …3d ago
-
Senior Robotics Controls Engineer USD 180K-200KActuator control | BLDC | C Sharp | C# | C++401k | Accident insurance | Commuter benefits | Company-Paid Holidays | Critical illness insuranceSenior-level Full TimeBrisbane, CA4d ago
-
A/B | A/B Testing | Analytics engineering | Application Performance Monitoring | Application performanceEquity package | Flexible leave options | Inclusive parental leave | Remote work within Australia | Wellbeing allowanceSenior-level Full TimeSydney, Australia R4d ago
-
Data and Analytics Support AUD 108K-120KADLS | Azure Data | Azure Data Lake | Azure Data Lake Storage | Data FactoryCareer development | Customer service focused benefits | Paid time to recharge | Part-time telework | Program to enhance cultureMid-level Full TimeBrisbane, QLD, AU4d ago
-
Data Engineer AUD 125K-126KAgile | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake StorageAdditional annual leave | Education assistance program | Employer funded insurance | Parental leave | Wellbeing leaveMid-level Full TimeSydney - Darling Park, Australia R4d ago
-
Machine Learning Platform Engineer SGD 139K-180KAWS | Azure | CI/CD | Dask | Distributed SystemsHealth and wellbeing resources | Paid volunteer daysMid-level Full TimeSydney - Australia - Sydney, 2000 … R4d ago
-
Senior Data Engineer - ATP AUD 153K-153KAgile | Azure DevOps | CI/CD | Confluence | Data ModellingAdditional leave | Career development opportunities | Discount on UNSW courses | Discounts on retail education and fitness passport | Flexible working optionsSenior-level Full TimeSydney, NSW R5d ago
-
HPC Systems Engineer AUD 90K-120KAnsible | C# | Cause analysis | Configuration Management | DebuggingMid-level Full TimeSydney5d ago