Data Engineer - Web Scraping
Tasks
- Build internal automation tools
- Clean transform validate large datasets
- Design web scraping systems
- Develop automated data ingestion pipelines
- Develop web scrapers for structured data sources
- Implement data quality control checks
- Schedule monitor optimize scraping workflows
- Troubleshoot data pipeline incidents
Perks/Benefits
- Collaborative team culture
- Continuous learning
- Flexible work environment
- High autonomy
- Professional growth
Skills/Tech-stack
APIs | AWS | AWS Lambda | Amazon RDS | Amazon S3 | Amazon SNS | Amazon SQS | Apache Airflow | CI/CD | Data Validation | Docker | GitLab CI | GitLab CI/CD | HTML | HTML parsing | JavaScript | Jenkins | Kubernetes | Pandas | Python | SQL | Scrapy | Selenium | XPath
Education
Bachelor of Engineering | Bachelor of Science | Master of Science
Roles
Related jobs
-
Senior-level Full TimeBengaluru, Karnataka, India5h ago
-
Data Engineer INR 3000K-4000KAWS | Amazon Aurora | Azure | Data Management | Data ModelingBike to work | Career development program | Educational assistance | Employee assistance programme | Employee discountSenior-level Full TimeBengaluru, KA, India6h ago
-
EGW Senior Embedded Lead INR 2200K-3000KBootloader | C++ | C++17 | CMake | Command LineHybrid workSenior-level Full TimeBangalore, Karnataka, India10h ago
-
Senior-level Full TimePune, India18h ago
-
Senior DevOps Engineer - Embedded Payments INR 3242K-4500KAPI Gateway | AWS | Aurora PostgreSQL | Bitbucket Pipelines | CloudWatchFlexible working arrangements | Mental health support | Paid parental leave | Volunteer daysSenior-level Full TimePune, MH, India R20h ago
-
AI Lead developer INR 3242K-4995KAgent-based | Agent-based systems | Agile methodologies | Azure Blob | Azure Blob StorageCareer advancement | Hybrid work options | International mobility | Learning and development | Professional developmentSenior-level Full TimeBengaluru21h ago
-
AWS | Amazon Kinesis | Amazon Redshift | Apache Airflow | Apache BeamFree games | Hybrid work | Learning and growth opportunities | Medical insurance | Mental health supportEntry-level Full Time地点：Hyderabad, Telangana, India21h ago
-
Senior-level Full TimeIN-AP-Hyderabad21h ago
-
AI | API Testing | Apache Airflow | Azure Databricks | CI/CDEmployee Assistance Program (EAP) | Flexible working environment | LinkedIn Learning | Volunteer time offSenior-level Full TimeChennai, TN, India21h ago
-
Mid-level Full TimeLocations: Hyderabad, Telangana, India22h ago
-
AWS CloudFormation | Amazon Kinesis | Amazon Redshift | Amazon Web Services | Apache AirflowCareer growth opportunities | Complimentary games | Family leave | Healthcare coverage | Mental wellbeing supportEntry-level Full TimeLocations: Hyderabad, Telangana, India22h ago
-
Senior Data Engineer - Internal INR 550K-850KAWS | Agile | Airflow | Amazon Kinesis | Amazon RedshiftEntry-level Full TimeAhmedabad, Gujarat, India22h ago
-
GBM - Public Dept-Bengaluru-Associate-Quantitative Engineering INR 1200K-2200KAutomation | C++ | Derivatives pricing | Equity Derivatives | Equity swapsMid-level Full TimeBengaluru, Karnataka, India22h ago
-
Lead AI/ML Engineer INR 3264K-5000KAWS | Anaconda | Azure | CI/CD | Computer VisionCareer growth opportunities | Fitness offerings | Flexible work environment | Mental Health plans | Remote-firstSenior-level Full TimeNoida23h ago
-
Staff Software Engineer, AI/ML and Moma Agentic INR 2500K-5000KArtificial Intelligence | C# | C++ | Database technologies | Distributed SystemsMentorshipSenior-level Full TimeHyderabad, Telangana, India23h ago
-
Senior Staff Software Engineer, Google Cloud Storage INR 3200K-4000KArtificial Intelligence | Benchmarking | C++ | Cloud Storage | Distributed SystemsSenior-level Full TimeHyderabad, Telangana, India23h ago
-
AI Customer Engineer, Public Sector, Google Cloud INR 2500K-5000KAPI Gateway | Audit Logging | Business automation | Cloud Networking | Distributed tracingConference participation | Travel opportunitiesSenior-level Full TimeGurugram, Haryana, India23h ago
-
GCP, Bigquery, Airflow/Consultant Specialist INR 1500K-2500KAgile | Airflow | Apache Kafka | Apache NiFi | BigQuerySenior-level Full TimeHyderabad, Telangana, India R1d ago
-
GCP, Airflow/CONSULTANT SPECIALIST INR 1500K-2500KAccess Control | Apache Airflow | Apache NiFi | Audit Logging | BigQueryFlexible working | Opportunities for professional developmentSenior-level Full TimePune, Maharashtra, India R1d ago
-
Senior Manager of Software Engineering Analytics Platform INR 3000K-5000KAgile | Artificial Intelligence | CI/CD | Cloud Native | DatabricksSenior-level Full TimeHyderabad, Telangana, India1d ago
-
Senior Software Engineer INR 2475K-5000K.NET | Agile | Artificial Intelligence | Azure Functions | Cloud servicesSenior-level Full TimeIndia, Karnataka, Bangalore; India, Telangana, Hyderabad1d ago
-
Senior Engineer - Operational Excellence (AI Solutions) INR 2500K-5000KAI Agent | AI Foundry | AI agent workflows | AI guardrails | APIsSenior-level Full TimePune, Maharashtra, IND, 4110131d ago
-
Senior-level Full TimeHyderabad, TS, India1d ago
-
Software Engineer III - Python, AI ML, Cloud INR 3500K-5500KAgentic Frameworks | Amazon Web Services | Apache Airflow | Automated testing | AzureSenior-level Full TimeMumbai, Maharashtra, India1d ago
-
Senior-level Full TimeHyderabad, Telangana, India1d ago