Senior Developer - Python

This role is for Weekday's client.

We are looking for a highly experienced Web Scraping Engineer with expertise in Python to support, maintain, and scale our extensive web scraping infrastructure. This role will focus on managing and optimising scraping scripts for over 200 websites, ensuring the delivery of accurate, timely data to fuel market analysis and competitive insights. The ideal candidate will have a strong background in Python, experience in web scraping at scale, and a commitment to reliable data collection and processing.

Requirements

Key Responsibilities

• Maintenance of Existing Scripts: Provide ongoing support and maintenance for existing Python web scraping scripts across 200+ websites, ensuring reliability and data accuracy. This can range from tweaking the existing script to redeveloping a new script based on target website changes. Partner with QA and Data Science and other teams.
• Script Development and Upkeep: Develop new scraping scripts as needed and update existing ones, ensuring timely and effective creation of new data sources to support business goals.
• Project Scaling: Assist in scaling the scraping project to cover additional competitive dealerships as required, optimising scripts to handle increased data load and expanding the reach of data collection.
• Daily Data Updates: Ensure daily data updates for all active scraping scripts, maintaining a steady flow of pricing, inventory details, and market analysis data for different RV models.

Requirements

• Strong proficiency in HTML, Python and Github with 5+ years experience
• Experience in web scraping frameworks (e.g., BeautifulSoup, Scrappy, Selenium). 
• Proven experience in maintaining and scaling web scraping projects.
• Experience with data storage formats (e.g., CSV, database, XML) and best practices in data management.
• Ability to troubleshoot and resolve issues within existing scripts and optimize for performance.
• Strong analytical skills, with the ability to adapt to dynamic data requirements.
• Experience in cloud-based data solutions for web scraping projects. 
• Knowledge of market analysis in a competitive business landscape, with a focus on pricing data and product inventory insights.