Resident Solution Architect Databricks
Sydney, New South Wales, Australia
AUD 140K-175K (estimate) Senior-level Contract Full Time
Tasks
- Automate pipeline deployment and CI CD
- Build and optimize ETL and ELT pipelines
- Build data quality validation frameworks
- Create PySpark reusable libraries
- Design and implement Data Lakehouse solutions
- Develop data ingestion frameworks
- Develop real-time streaming solutions
- Enable analytics and machine learning workflows
- Establish data engineering standards and documentation
- Implement alerting and logging
- Implement data governance and access control
- Mentor junior engineers
- Migrate legacy data sources to cloud
- Monitor pipeline performance and troubleshoot issues
- Optimize data storage and query performance
- Translate business requirements into scalable data solutions
Perks/Benefits
- N/A
Skills/Tech-stack
Apache Spark | Azure | Azure Data | Azure Data Factory | Azure DevOps | Azure Event | Azure Event Hubs | Azure Key Vault | Batch Processing | CI/CD | Configuration Management | Data Factory | Data Governance | Data Ingestion | Data Quality | Data Security | Databricks | Databricks Structured Streaming | Delta Lake | Event Hubs | Git | Great Expectations | Kafka | Key Vault | Least Privilege | MLflow | Parquet | PySpark | Streaming Data | Structured Streaming | Unity Catalog
Education
N/A
Roles
Regions
Countries
States
Cities
Related jobs
-
Data Architect AUD 125K-135KAWS | Data Governance | Data Modelling | Data integration | Enterprise ArchitectureAdditional leave days | Flexible work arrangements | Recognition program | Volunteer leaveSenior-level Full TimeMelbourne, VIC, AU2d ago
-
AI | API Gateway | AWS | AgentCore) | AzureFlexible work arrangements | Market-leading parental leave | Paid volunteering days | Retirement plan | Wellbeing leaveMid-level Full TimeSydney, NSW, AU3d ago
-
Solutions Architect - Storage AUD 119K-200KAWS | Active Listening | Azure | Block Storage | CRM systemsHealth & Wellbeing | Inclusive environment | Professional developmentSenior-level Full TimeCanberra, Australian Capital Territory, Australia3d ago
-
Solutions Architect - Storage AUD 119K-200KAWS | Azure | Block Storage | Cloud Platforms | Data MigrationCareer growth programs | Health benefits | Inclusive culture | Professional developmentSenior-level Full TimeCanberra, Australian Capital Territory, Australia3d ago
-
Business AI Architect AUD 100K-140KAIBusinessServices | AICore | APIs | Business Process | Business Process AnalysisFlexible work arrangements | Inclusive work environment | Learning and development opportunitiesSenior-level Full TimeNorth Sydney, NSW, AU, 20609d ago
-
Assistant Vice President - Enterprise Data Architect AUD 125K-150KCloud Data | Cloud Data Platforms | Data Architecture | Data Governance | Data QualityAI certifications | Career growth opportunities | Inclusive culture | Mentorship | Training programsSenior-level Full TimeAustralia-Sydney16d ago
-
Manager, Professional Services AUD 112K-120KAI | AI Engineering | Apache Spark | Data Migration | Data platformFlexible work arrangements | Inclusive culture | Professional developmentMid-level Full TimeSydney, Australia16d ago
-
Relocate to Malta Data Architect (Intelligent Cloud/ Data & Analytics/ Consulting/Big4) A AUD 120K-155KAzure Blob | Azure Blob Storage | Azure Cosmos | Azure Cosmos DB | Azure DataCareer growth opportunities | Relocation support | Training and developmentSenior-level Full TimeCanberra, ACT, Australia18d ago
-
AWS Consultant AUD 125K-160KAWS | Apache Spark | Avro | CI/CD | Data ModelingCollaborative environment | Flexible schedulesSenior-level Contract Full TimeMelbourne, Victoria, Australia18d ago
-
AI Architect AUD 110K-120KAI Data | AI Evaluation | AI Observability | AI data architecture | AI evaluation frameworksChill-out spaces | Flexi hours | Professional development | Remote Working | Social eventsSenior-level Full TimeSydney, New South Wales, Australia21d ago
-
Senior Data Archiect | Mandarin Speaking AUD 145K-180KClickHouse | Data Architecture | Data Governance | Data Modeling | Data Pipeline DevelopmentCareer growth opportunities | Flexible work arrangements | Global work environment | Health benefitsSenior-level Full TimeSydney, New South Wales, Australia25d ago