Senior Data Engineer - Web Scraping
Tasks
- Automate web scraping processes
- Build data pipelines
- Clean and transform datasets
- Design data products
- Design web scrapers
- Develop and maintain scrapers
- Develop automation tools
- Explore and prepare datasets
- Implement alerting systems
- Implement quality control checks
- Ingest scraped data into data warehouses
- Ingest scraped data into databases
- Investigate data incidents
- Orchestrate scheduled workflows
- Resolve operational issues
- Support data scientists with tools
- Validate data accuracy
- Validate data availability
- Validate data consistency
- Validate data integrity
Perks/Benefits
- Autonomy
- Collaborative team
- Flexible location
- Fully remote
- Operational excellence focus
- Ownership
- Professional innovation focus
Skills/Tech-stack
APIs | AWS | Apache Airflow | CI/CD | Data Pipelines | Data Warehousing | Docker | GitHub Actions | HTML | JavaScript | Jenkins | Kubernetes | Lambda | Pandas | Postman | Python | RDS | S3 | SNS | SQL | SQS | Scrapy | Selenium | XPath
Education
Bachelor of Engineering | Bachelor of Science | Master of Science
Roles
Related jobs
-
Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure Data Lake Storage Gen2Flexible work location | Fully remoteSenior-level Contract Full TimeAustralia R23h ago
-
Data Engineer AUD 120K-126KCode review | Data Modeling | Data Warehousing | Data integration | Data pipelineDiscounted air travel | Discounted lounge membership | Discounts on travel insurance and car hire | Dress for your day | Employee assistance programMid-level Full TimeBrisbane, Melbourne, Sydney R1d ago
-
Forward Deployed Engineer - Sydney AUD 116K-131KCode review | Full Stack | Full-Stack Development | Generative AI | JavaScriptHybrid work | Relocation assistance | Travel up to 50 percentMid-level Full TimeSydney, Australia R1d ago
-
Senior Data Engineer AUD 129K-142KAWS | Automation | Cybersecurity | Data Governance | Data ModellingFlexible work options | Hybrid work | Mentoring cultureSenior-level Full TimeMacquarie Park, NSW, AU, 2113 R2d ago
-
Senior Machine Learning Engineer (Platform / MLOPs) AUD 132K-180KAPI Development | AWS | Amazon EKS | ArgoCD | Arize AIAnnual flu vaccinations | Gym access | Hybrid flexibility | In-office lunch | Nearmap subscriptionSenior-level Full TimeBarangaroo, NSW, Australia R3d ago
-
Robotics Software Engineer AUD 401KBenchmarking | C++ | CI/CD | Collision Avoidance | Machine Learning401k matching | Dental insurance | Health insurance | Paid Holidays | Travel supportMid-level Full TimeCharlestown, MA R3d ago
-
Senior LLMOps Engineer AUD 147K-180KAlerting | Dashboarding | Data Modeling | Data Pipelines | DatadogEquity | Fertility support | Health and wellness allowance | Home office budget | Learning and development budgetSenior-level Full TimeMelbourne R4d ago
-
Forward Deployed Engineer GBP 25K-25K100GbE | C++ | Distributed Systems | Docker | File systemsOn-site work option | Travel up to 25 percentSenior-level Full TimeAustralia, AU R4d ago
-
Computer Vision | Deep learning | Diffusion Models | Docker | Generative AIEquity package | Flexible leave options | Flexible work arrangements | Parental leave | Professional growth opportunitiesSenior-level Full TimeAustralia R7d ago
-
AI Engineer AUD 132K-180KAWS | Agent Development | Amazon Bedrock | Artificial Intelligence | AutomationFlexible hours | Free food and snacks | Opportunity to drive projects | Socials and Community Groups | Training and developmentSenior-level Full TimeMelbourne, Victoria, Australia R7d ago
-
AWS | AWS CloudFormation | AWS Glue | AWS Lambda | AWS Step FunctionsFlexible work location | Fully remote | Ownership and leadership opportunities | Technical autonomySenior-level Contract Full TimeAustralia R7d ago
-
Senior Machine Learning Engineer AUD 144K-180KAWS | Algorithms | Anomaly Detection | CI/CD | Cloud platformClassPass membership | Culture Swap Days | Employee resource groups | Extra annual leave | Flexible work hoursSenior-level Full TimeSydney, Australia R8d ago
-
Infrastructure Engineer, APAC AUD 92K-127KAWS | Ansible | Bash | CI/CD | DockerCompany-provided equipment | Growth opportunities | Happy hours | Internet reimbursement | Learning opportunitiesMid-level Full TimeAdelaide (Hybrid) R8d ago
-
AI Engineering Lead AUD 137K-143KAgile | Amazon Web Services | Docker | IAM | LoggingDiscounted flights | Discounts on accommodation | Discounts on car hire | Discounts on experiences | Discounts on lounge membershipSenior-level Full TimeBrisbane, Sydney R8d ago
-
Forward Deployed AI Engineer, Australia AUD 119K-161KAWS | Agentic Workflows | Apache Iceberg | Apache NiFi | Apache SparkContinued Career Development | Employee resource groups | Flexible WFH | Generous PTO | Mental and physical wellness programsSenior-level Full TimeAustralia-Sydney-Remote R11d ago
-
Senior Data Engineer AUD 150K-170KA/B | A/B Testing | AWS | Amplitude | B testingAnnual leave | Catered lunch | Equity and ownership | Flexible time off | Hybrid work modeSenior-level Full TimeSydney, New South Wales R14d ago
-
Senior Analyst, Data & Analytics AUD 60K-78KBusiness Intelligence | DBT | Data Governance | Data Modeling | Data PipelinesEmployee discount travel | Hybrid work model | Staff travel programSenior-level Full TimeMelbourne, Victoria, Australia R15d ago
-
Data Analysis | Machine Learning | Model Evaluation | Model Serving | NumPyEquity packages | Flexible leave options | Inclusive parental leave | Virtual interviews | Wellbeing allowanceSenior-level Full TimeSydney, Australia R16d ago
-
Senior Data Engineer - Automation & AI AUD 118K-127KAccess Control | Agentic Systems | Auditability | Batch Processing | ChunkingEmployee assistance program | Employee wellbeing platform | Extra days leave | Flexibility support | Hybrid work modelSenior-level Full TimeAustralia R20d ago
-
Senior Machine Learning Engineer AUD 148K-210KAccess Control | Agent systems | CI/CD | Capabilities Based Access Control | Error RecoveryHealth and wellness benefits | Hybrid work schedule | Remote work flexibilitySenior-level Full TimeAustralia - Melbourne - Hybrid R23d ago
-
Staff Machine Learning Engineer AUD 140K-185KAWS | Amazon ECS | Amazon ElastiCache | Amazon S3 | Audit LoggingSenior-level Full TimeAustralia - Melbourne - Hybrid R23d ago
-
Senior Machine Learning Engineer AUD 145K-186KAI Risk Management Framework | Agent systems | Agentic Workflows | Anomaly Detection | Content SafetyHybrid work model | Inclusive work environment | Reasonable accommodations | Remote work flexibilitySenior-level Full TimeAustralia - Melbourne - Hybrid R23d ago
-
A/B | A/B Testing | B testing | Cost Optimization | Data PipelinesEquity packages | Flexible leave | Parental leave | Remote within Australia | Wellbeing allowanceSenior-level Full TimeMelbourne, VIC, Australia R24d ago
-
Amazon Web Services | Apache Spark | Big Data | Data Analysis | HadoopRemote workSenior-level Full TimeSydney, NSW, Australia R27d ago
-
Senior Analytics Engineer - User Voice AUD 148K-170KAlerting | Analytics engineering | Census | DBT | Data GovernanceEquity packages | Flexible leave options | Hybrid work | Inclusive parental leave | Wellbeing allowanceSenior-level Full TimeMelbourne, Australia R1mo ago