Resident Solution Architect Databricks
Sydney, New South Wales, Australia
AUD 140K-175K (estimate) Senior-level Contract Full Time
Tasks
- Automate pipeline deployment and CI CD
- Build and optimize ETL and ELT pipelines
- Build data quality validation frameworks
- Create PySpark reusable libraries
- Design and implement Data Lakehouse solutions
- Develop data ingestion frameworks
- Develop real-time streaming solutions
- Enable analytics and machine learning workflows
- Establish data engineering standards and documentation
- Implement alerting and logging
- Implement data governance and access control
- Mentor junior engineers
- Migrate legacy data sources to cloud
- Monitor pipeline performance and troubleshoot issues
- Optimize data storage and query performance
- Translate business requirements into scalable data solutions
Perks/Benefits
- N/A
Skills/Tech-stack
Apache Spark | Azure | Azure Data | Azure Data Factory | Azure DevOps | Azure Event | Azure Event Hubs | Azure Key Vault | Batch Processing | CI/CD | Configuration Management | Data Factory | Data Governance | Data Ingestion | Data Quality | Data Security | Databricks | Databricks Structured Streaming | Delta Lake | Event Hubs | Git | Great Expectations | Kafka | Key Vault | Least Privilege | MLflow | Parquet | PySpark | Streaming Data | Structured Streaming | Unity Catalog
Education
N/A
Roles
Regions
Countries
States
Cities
Related jobs
-
Solutions Architect AUD 147K-182KApache Spark | Big Data | Big data analytics | Cloud platform | Data analyticsSenior-level Full TimeAustralian Capital Territory, Australia1d ago
-
Senior-level Full TimeSydney Olympic Park, New South Wales, …3d ago
-
Chief Engineer - Data Platforms AUD 60K-200KAgile | Apache Airflow | Apache Kafka | Artificial Intelligence | AzureAdditional leave options | Employee discounts | Flexible work arrangements | Tailored learning and development | Volunteering opportunitiesExecutive-level Full TimeSydney, NSW, Australia8d ago
-
Data Engineering Lead AUD 129K-175KAutomated testing | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake StorageSenior-level Full TimeHJ Head Office, Australia8d ago
-
Solutions Architect AUD 147K-185KApache Spark | Big Data | Big data analytics | Cloud platform | Data analyticsSenior-level Full TimeQueensland, Australia9d ago
-
AI Architect - Databricks AUD 140K-180KAmazon Web Services | Azure | CI/CD | Compliance | Data GovernanceSenior-level Contract Full TimeSydney, New South Wales, Australia11d ago
-
AI Solution Architect - Advisory AUD 60K-140KAI Search | Artificial Intelligence | Azure AI | Azure AI Search | Azure DataFlexible working hours | Flexible working locations | Health and wellbeing initiatives | Learning and growth opportunities | Paid parental leaveSenior-level Full TimeSydney, NSW, Australia14d ago
-
AI Solution Architect - Advisory AUD 60K-140KAI Search | Artificial Intelligence | Azure AI | Azure AI Search | Azure DataCareer breaks | Flexible working hours | Flexible working locations | Generous paid parental leave | Health and wellbeing initiativesSenior-level Full TimeSydney, NSW, Australia15d ago
-
Delivery Solutions Architect AUD 147K-209KApache Spark | Business Analysis | Data Engineering | Data analytics | Delta LakeHybrid work | Travel up to 30 percentSenior-level Full TimeBrisbane, Australia; Melbourne, Australia15d ago
-
Lead Data & AI Architect AUD 147K-195KAWS Redshift | Agent Orchestration | Apex | BigQuery | Cloud ComputingAdoption Assistance | Fertility benefits | Parental leave | Wellbeing reimbursementSenior-level Full TimeAustralia - Sydney15d ago
-
AI Architect AUD 129K-178KAI architecture | AWS | AWS Bedrock | Adversarial risk | Adversarial risk mitigation360° feedback | Innovation days | Mentoring support | Paid certifications | Structured development pathwaysSenior-level Full TimeSydney, New South Wales, Australia15d ago
-
Artificial Intelligence - Architect AUD 120K-135KAWS | Apache Kafka | Apache Spark | Azure | Computer VisionCooperative work environment | Diversity and inclusion | Global culture | Training opportunitiesMid-level Contract Full TimeSydney, New South Wales, Australia15d ago
-
Amazon Web Services | Analytics Cloud | Apache Hadoop | Apache Spark | Artificial IntelligenceContinuous learning | Flexible working models | Health and wellbeing focusSenior-level Full TimeNorth Sydney, NSW, AU, 206018d ago
-
Sr. Solutions Engineer AUD 150K-205KApache Spark | Cloud platform | Data Architecture | Delta Lake | JavaSenior-level Full TimeBrisbane, Australia20d ago
-
Solutions Architect (Adelaide) AUD 170K-230KApache Spark | Big Data | Big data analytics | Cloud Platforms | Data analyticsSenior-level Full TimeRemote - Australia R20d ago
-
Data Architect AUD 125K-135KAWS | Data Governance | Data Modelling | Data integration | Enterprise ArchitectureAdditional leave days | Flexible work arrangements | Recognition program | Volunteer leaveSenior-level Full TimeMelbourne, VIC, AU23d ago
-
AI | API Gateway | AWS | AgentCore) | AzureFlexible work arrangements | Market-leading parental leave | Paid volunteering days | Retirement plan | Wellbeing leaveMid-level Full TimeSydney, NSW, AU24d ago
-
Solutions Architect - Storage AUD 119K-200KAWS | Active Listening | Azure | Block Storage | CRM systemsHealth & Wellbeing | Inclusive environment | Professional developmentSenior-level Full TimeCanberra, Australian Capital Territory, Australia24d ago
-
Solutions Architect - Storage AUD 119K-200KAWS | Azure | Block Storage | Cloud Platforms | Data MigrationCareer growth programs | Health benefits | Inclusive culture | Professional developmentSenior-level Full TimeCanberra, Australian Capital Territory, Australia24d ago
-
Business AI Architect AUD 100K-140KAIBusinessServices | AICore | APIs | Business Process | Business Process AnalysisFlexible work arrangements | Inclusive work environment | Learning and development opportunitiesSenior-level Full TimeNorth Sydney, NSW, AU, 206030d ago
-
Assistant Vice President - Enterprise Data Architect AUD 125K-150KCloud Data | Cloud Data Platforms | Data Architecture | Data Governance | Data QualityAI certifications | Career growth opportunities | Inclusive culture | Mentorship | Training programsSenior-level Full TimeAustralia-Sydney1mo ago
-
Manager, Professional Services AUD 112K-120KAI | AI Engineering | Apache Spark | Data Migration | Data platformFlexible work arrangements | Inclusive culture | Professional developmentMid-level Full TimeSydney, Australia1mo ago
-
Relocate to Malta Data Architect (Intelligent Cloud/ Data & Analytics/ Consulting/Big4) A AUD 120K-155KAzure Blob | Azure Blob Storage | Azure Cosmos | Azure Cosmos DB | Azure DataCareer growth opportunities | Relocation support | Training and developmentSenior-level Full TimeCanberra, ACT, Australia1mo ago
-
AWS Consultant AUD 125K-160KAWS | Apache Spark | Avro | CI/CD | Data ModelingCollaborative environment | Flexible schedulesSenior-level Contract Full TimeMelbourne, Victoria, Australia1mo ago
-
AI Architect AUD 110K-120KAI Data | AI Evaluation | AI Observability | AI data architecture | AI evaluation frameworksChill-out spaces | Flexi hours | Professional development | Remote Working | Social eventsSenior-level Full TimeSydney, New South Wales, Australia1mo ago