Harvest – Python Web Scraping Toolkit

Harvest packages the hard parts of reliable scraping — retries, throttling, proxies and monitoring — into a reusable toolkit.

Key features

• Playwright and HTTPX crawlers
• Proxy rotation and rate limiting
• Cron scheduling and job dashboard
• Exports to CSV, PostgreSQL and S3
• Schema validation of extracted records

Tech stack

• Python 3.12
• Playwright
• FastAPI dashboard

Requirements

• Python 3.11+

Installation

1. Create a virtual environment with Python 3.11+.
2. Run pip install -r requirements.txt.
3. Copy .env.example to .env and set your credentials.
4. Run python manage.py migrate (or the included CLI) and start the service.

What's included

• Full source code with comments
• Step-by-step documentation
• Demo data and seed scripts where applicable
• 6 months of author support and all future updates