Senior Data Engineer - Web Scraping
Tasks
- Build automation for web scraping platform
- Collaborate with stakeholders to translate requirements
- Design web scrapers
- Develop & maintain data pipelines
- Develop tools and applications for data products
- Implement data quality control checks
- Ingest data into databases data warehouses
- Manage workflows with Apache Airflow
- Set up alerting systems and incident response
- Transform clean large datasets
Perks/Benefits
Skills/Tech-stack
APIs | AWS | AWS Lambda | Amazon RDS | Amazon S3 | Amazon SNS | Amazon SQS | Apache Airflow | CI/CD | Docker | Fiddler | GitHub Actions | HTML | JavaScript | Jenkins | Kubernetes | Pandas | Postman | Python | SQL | Scrapy | Selenium | XPath
Education
Bachelor of Engineering | Bachelor of Science | Master of Science
Roles
Related jobs
-
Databricks and Splunk Engineer - Senior Associate INR 3264K-4500KAmazon Web Services | Analytics | Azure | Cloud Computing | Cyber ThreatEmployee networks | Flexible work/life support | Hybrid work schedule | Inclusive development opportunities | Paid volunteer daysSenior-level Full TimeBangalore, India R11h ago
-
AI Integration Engineer – Digital Transformation (m/f/d) INR 2500K-3584KABAC | API Gateway | AWS Glue | AWS Lambda | AWS Secrets1 to 1 counselling | Discount vouchers | Hybrid work model | Language classes | Learning and developmentSenior-level Full TimeGurugram R18h ago
-
Data Engineer INR 3000K-4000KAWS | Amazon Aurora | Automation | Azure | Data ManagementBike to work | Career development program | Educational assistance | Employee assistance program | Employee discountSenior-level Full TimeBengaluru, KA, India R22h ago
-
Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure Data Lake Storage Gen2Autonomy | Fully remote | Remote work flexibilitySenior-level Contract Full TimeIndia R1d ago
-
AWS | Azure | Django | Embeddings | FastAPIContract or consulting engagement | Part-time engagement | Remote work within India | Technical ownership and autonomyMid-level Full TimeIndia R1d ago
-
AI Engineer INR 450K-700KData Preprocessing | Deep learning | Language Processing | MLOps | Machine LearningCareer development | Hybrid work flexibilityExecutive-level Full TimeNoida, India R1d ago
-
Staff Software Engineer, Machine Learning INR 3000K-4580KC++ | Classification | Data Pipelines | Deep learning | EmbeddingsHealth and wellness programs | Time away for employeesSenior-level Full TimeBengaluru, KA, India R1d ago
-
Senior Software Engineer, Data Platform INR 2000K-4200KApache Pinot | Artifactory | Git | Go | JavaSenior-level Full TimeIndia (Remote) R1d ago
-
AI/ML Engineering Lead/(Gen AI,Python,LLM,ETL)Associate Director INR 2000K-4000KApache Airflow | Apache Beam | Auditability | Automated testing | BigQueryContinuous professional development | Flexible working | Inclusive diverse environment | Opportunities for growthMid-level Full TimePune, Maharashtra, India R1d ago
-
API | AWS SageMaker | Azure Machine Learning | CI/CD | Context engineeringAdvanced AI exposure | Collaborative environment | Continuous learning | Contract engagement | End-to-end ownershipSenior-level Contract Full TimeIndia R1d ago
-
Azure | Azure AD | Azure Data | Azure Data Factory | Azure Data LakeFlexible work arrangements | Healthcare benefits | Insurance | Paid time off | Professional training and developmentSenior-level Full TimeIndia R1d ago
-
Senior Agentic Engineer, AI Platform INR 3000K-5000KAPI Development | AWS | Agent Orchestration | Authentication | AutoscalingSenior-level Full TimeBengaluru R1d ago
-
SW Engineer (1 - 2 years of exp in .Net / Python, Full-stack development, Angular / React, GenAI) INR 1500K-2500K.NET | API Management | Agile | Angular | Azure DevOpsHybrid work schedule | Limited direction working environmentMid-level Full TimeBengaluru, India R1d ago
-
Senior-level Full TimeHyderabad, India R1d ago
-
Senior-level Full TimeIndia - Remote R1d ago
-
Senior AI Engineer INR 2475K-3465KAirflow | Amazon Web Services | Angular | Apache Beam | BigQueryCareer development | Healthcare benefits | Hybrid work | Learning platform | Paid time offSenior-level Full TimeIND-Pune-Equifax Analytics-PEC, India R1d ago
-
Lead AI Engineer INR 2156K-3500KAWS | Agile | Apache Airflow | Apache Beam | Atlassian ConfluenceGuided career tracks | Healthcare benefits | Hybrid work setting | Online learning platform | Paid time offSenior-level Full TimeIND-Pune-Equifax Analytics-PEC, India R1d ago
-
Cloud/Big Data Engineer INR 2771K-3900KAgile | Apache Hive | Apache Spark | Automation | BigQueryHybrid work schedule | Work-life flexibilitySenior-level Full TimeChennai, India R1d ago
-
Lead Cloud Data Engineer - Manager INR 2771K-4064KAWS | Airflow | Autosys | CICD | Cloud SecurityEmployee networks | Flexible work/life support | Hybrid work schedule | Inclusive development opportunities | Paid volunteer daysSenior-level Full TimeHyderabad, India R1d ago
-
Architect - Data Engineer INR 2000K-4000KAWS Athena | AWS ECS | AWS EKS | AWS EMR | AWS GlueHybrid work schedule | In person collaboration at assigned office location minimum two days per week | Work-life flexibilitySenior-level Full TimeHyderabad, India R1d ago
-
Senior-level Full TimeBangalore, Karnātaka, India R1d ago
-
Senior-level Full TimeBangalore, Karnātaka, India R1d ago
-
Engineer, Java & Spark INR 2771K-4064KAgile | Amazon Web Services | Apache Spark | Automated testing | CI/CDHybrid workMid-level Full TimeChennai, India R1d ago
-
Principal Engineer - Agentic Engineering INR 2156K-3500KAgile | Airflow | Apache Beam | BigQuery | BigtableCareer growth | Healthcare benefits | Hybrid work environment | Online learning platform | Paid time offSenior-level Full TimeIND-Pune-Equifax Analytics-PEC, India R1d ago
-
Senior-level Full TimePune, India R1d ago