Senior Data Engineer - Web Scraping
Tasks
- Build automation tools
- Collaborate with stakeholders to define requirements
- Design web scrapers
- Develop data pipelines
- Implement data quality checks
- Ingest data into databases and data warehouses
- Monitor alerts and resolve incidents
- Orchestrate workflows with Apache Airflow
- Transform scraped data
Perks/Benefits
Skills/Tech-stack
APIs | AWS Lambda | AWS S3 | Amazon RDS | Amazon SNS | Amazon SQS | Apache Airflow | CI/CD | Docker | GitHub Actions | HTML | JavaScript | Jenkins | Kubernetes | Pandas | Python | SQL | Scrapy | Selenium | XPath
Education
Bachelor of Engineering | Bachelor of Science | Master of Science
Roles
Related jobs
-
AI code generation | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake StorageFully remote | Remote flexibility | Work autonomySenior-level Contract Full TimeSouth Africa R23h ago
-
Data Engineer ZAR 420K-464KAPI | Artificial Intelligence | DBT | Data Compliance | Data GovernanceCollaborative culture | Employee wellness programs | Hybrid work | Modern tooling | Professional developmentMid-level Full TimeCape Town, South Africa R1d ago
-
AWS | Autogen | Azure | Claude | Cloud platformBackup power required | Fixed internet required | Public holiday compensation | Remote workSenior-level Full TimeSouth Africa - Remote R2d ago
-
AI Evaluation | API Integration | AWS | Access Control | Agent OrchestrationFlexible work hours | Fully remoteSenior-level Full TimeEgypt R3d ago
-
Lead Data and AI Integration Specialist ZAR 540K-720KAWS | Anomaly Detection | Authentication | Automation | AzureEqual opportunity workplace | Hybrid working model | Travel to primary office as requiredSenior-level Full Time15 Alice Lane, South Africa R7d ago
-
AWS | AWS CloudFormation | AWS Glue | AWS Glue Catalog | AWS LakeAgile collaboration | Flexible work location | Fully remote | Technical autonomySenior-level Contract Full TimeSouth Africa R7d ago
-
Full Stack AI Developer ZAR 540K-540KAI workflow | AI workflow automation | Anthropic API | Cloud deployment | DockerFull-time remote work | US business hoursMid-level Full TimeSouth Africa - Remote R11d ago
-
Data Engineer ZAR 396K-456KAPI Development | AWS | Apache Spark | Automation | CI/CDCareer development opportunities | Employee discounts | Hybrid working | Paid sabbaticals | Paid time offSenior-level Full TimeCape Town, Western Cape, South Africa R1mo ago
-
Principal Clinical Data Engineer - based in South Africa ZAR 663K-663K21 CFR | 21 CFR Part 11 | CFR Part 11 | Cause analysis | Clinical dataSenior-level Full TimeSouth Africa-Bloemfontein-Remote R1mo ago
-
Airflow | Amazon Redshift | BigQuery | CI/CD | ChromaDBFlexible working hours | Remote workSenior-level Full TimeSouth Africa - Remote R1mo ago
-
Senior Clinical Data Engineer ZAR 663K-663K21 CFR | 21 CFR Part 11 | CFR Part 11 | Cause analysis | Clinical ResearchSenior-level Full TimeSouth Africa-Bloemfontein-Remote R1mo ago
-
AWS | Amazon Redshift | Apache Airflow | Azure | BigQueryCareer growth opportunities | Collaborative culture | Continuous learning | Flexible working arrangements | Fully remoteSenior-level Full TimeSouth Africa - Remote R1mo ago