Resident Solution Architect Databricks
Sydney, New South Wales, Australia
AUD 140K-175K (estimate) Senior-level Contract Full Time
Tasks
- Automate pipeline deployment and CI CD
- Build and optimize ETL and ELT pipelines
- Build data quality validation frameworks
- Create PySpark reusable libraries
- Design and implement Data Lakehouse solutions
- Develop data ingestion frameworks
- Develop real-time streaming solutions
- Enable analytics and machine learning workflows
- Establish data engineering standards and documentation
- Implement alerting and logging
- Implement data governance and access control
- Mentor junior engineers
- Migrate legacy data sources to cloud
- Monitor pipeline performance and troubleshoot issues
- Optimize data storage and query performance
- Translate business requirements into scalable data solutions
Perks/Benefits
- N/A
Skills/Tech-stack
Apache Spark | Azure | Azure Data | Azure Data Factory | Azure DevOps | Azure Event | Azure Event Hubs | Azure Key Vault | Batch Processing | CI/CD | Configuration Management | Data Factory | Data Governance | Data Ingestion | Data Quality | Data Security | Databricks | Databricks Structured Streaming | Delta Lake | Event Hubs | Git | Great Expectations | Kafka | Key Vault | Least Privilege | MLflow | Parquet | PySpark | Streaming Data | Structured Streaming | Unity Catalog
Education
N/A
Roles
Regions
Countries
States
Cities
Related jobs
-
Solution Architect Data Warehousing AUD 149K-173KBusiness Intelligence | Cloud Computing | Cloud Migration | Data Architecture | Data GovernanceAutonomous work environment | Employee assistance program | Flexible working environment | Health and wellbeing programs | Training and development opportunitiesSenior-level Full TimeParramatta, NSW, AU, 21501d ago
-
Data Architect AUD 125K-135KAWS | Azure | Cost Optimization | DBT | Data EngineeringAnnual wellness allowance | Charitable Volunteering Time | Health insurance subsidy | Ongoing training | Superannuation contributionsSenior-level Full TimeSydney, New South Wales, Australia3d ago
-
Expert Data Governance and Data Products AUD 129K-142KAI machine learning | Data Architecture | Data Governance | Data Management | Data Product ManagementEqual employment opportunities | Flexible working arrangements | Paid parental leave | Training opportunitiesSenior-level Full TimeNorth Sydney, NSW, Australia6d ago
-
Solutions Architect, Google Forge, Google Cloud AUD 147K-185KAI Model Life Cycle | AI model | CI/CD | Cloud platform | Data analyticsSenior-level Full TimeSydney NSW, Australia; Docklands VIC, Australia8d ago
-
Resident Solutions Architect AUD 147K-182KApache Spark | Cloud Computing | Data Engineering | Data Lakes | Data ScienceSenior-level Full TimeWestern Australia, Australia9d ago
-
AI Architect AUD 90K-110KAI Foundry | AI Governance | AKS | API Design | AzureEmployee growth and development | Flexible hybrid working | Parental leave support | Work from home flexibilitySenior-level Full TimeAU PER Perth Office, Australia9d ago
-
Principal Data Architect AUD 139K-161KData Architecture | Data Governance | Data Modelling | Digital Twin | Enterprise DataFlexible working arrangements | Inclusion and diversity programs | Paid parental leave | Training opportunitiesSenior-level Full TimeVIC - Melbourne - 727 Collins …10d ago
-
Senior-level Full TimeMelbourne, Victoria, Australia11d ago
-
Solution Architect AUD 150K-188KArtificial Intelligence | CI/CD | Cloud Architecture | Data Governance | Data LakeSenior-level Full TimeSydney, Melbourne11d ago
-
Senior Data Architect (Enterprise) AUD 125K-135KCloud Computing | Data Architecture | Data Governance | Data Management | Data ModelingInclusive workplaceSenior-level Full TimeMelbourne Support Centre15d ago
-
Business AI Architect AUD 100K-150KAI Business Services | AI Core | AI-Hub | API Design | Analytics CloudSenior-level Full TimeSouthbank (Melbourne), VIC, AU, 300615d ago
-
Senior-level Full TimeSouthbank (Melbourne), VIC, AU, 300615d ago
-
AWS | Advanced Analytics | Analytics Maturity | Azure | BigQuery360° feedback | Innovation days | Paid certifications | Weekly TrainingSenior-level Full TimeMelbourne, Victoria, Australia15d ago
-
AI Architect - Fabric AUD 120K-140KAI Services | Access Management | Azure AI | Azure AI Services | Azure Data360-degree feedback | Community clubs | Innovation days | Mentorship | Paid certificationsSenior-level Full TimeSydney, New South Wales, Australia18d ago
-
Solutions Architect AUD 147K-182KApache Spark | Big Data | Big data analytics | Cloud platform | Data analyticsSenior-level Full TimeAustralian Capital Territory, Australia22d ago
-
Senior-level Full TimeSydney Olympic Park, New South Wales, …24d ago
-
Solutions Architect AUD 147K-185KApache Spark | Big Data | Big data analytics | Cloud platform | Data analyticsSenior-level Full TimeQueensland, Australia1mo ago
-
AI Architect - Databricks AUD 140K-180KAmazon Web Services | Azure | CI/CD | Compliance | Data GovernanceSenior-level Contract Full TimeSydney, New South Wales, Australia1mo ago
-
AI Solution Architect - Advisory AUD 60K-140KAI Search | Artificial Intelligence | Azure AI | Azure AI Search | Azure DataFlexible working hours | Flexible working locations | Health and wellbeing initiatives | Learning and growth opportunities | Paid parental leaveSenior-level Full TimeSydney, NSW, Australia1mo ago
-
AI Solution Architect - Advisory AUD 60K-140KAI Search | Artificial Intelligence | Azure AI | Azure AI Search | Azure DataCareer breaks | Flexible working hours | Flexible working locations | Generous paid parental leave | Health and wellbeing initiativesSenior-level Full TimeSydney, NSW, Australia1mo ago
-
Delivery Solutions Architect AUD 147K-209KApache Spark | Business Analysis | Data Engineering | Data analytics | Delta LakeHybrid work | Travel up to 30 percentSenior-level Full TimeBrisbane, Australia; Melbourne, Australia1mo ago
-
Lead Data & AI Architect AUD 147K-195KAWS Redshift | Agent Orchestration | Apex | BigQuery | Cloud ComputingAdoption Assistance | Fertility benefits | Parental leave | Wellbeing reimbursementSenior-level Full TimeAustralia - Sydney1mo ago
-
AI Architect AUD 129K-178KAI architecture | AWS | AWS Bedrock | Adversarial risk | Adversarial risk mitigation360° feedback | Innovation days | Mentoring support | Paid certifications | Structured development pathwaysSenior-level Full TimeSydney, New South Wales, Australia1mo ago
-
Artificial Intelligence - Architect AUD 120K-135KAWS | Apache Kafka | Apache Spark | Azure | Computer VisionCooperative work environment | Diversity and inclusion | Global culture | Training opportunitiesMid-level Contract Full TimeSydney, New South Wales, Australia1mo ago
-
Amazon Web Services | Analytics Cloud | Apache Hadoop | Apache Spark | Artificial IntelligenceContinuous learning | Flexible working models | Health and wellbeing focusSenior-level Full TimeNorth Sydney, NSW, AU, 20601mo ago