Proxies & blocked sites

Some sites (e.g. eBay, Instagram, Twitter/X, Spotify) block direct automated requests with anti-bot systems. PyScrappy supports two ways through them.

A proxy (or rotating list)

Applies to both the HTTP and browser backends:

from pyscrappy import ScraperConfig, AmazonScraper
 
# Single proxy
config = ScraperConfig(proxy="http://user:pass@host:port")
 
# Rotating list (one picked per request)
config = ScraperConfig(proxy=["http://p1:8080", "http://p2:8080"])
 
with AmazonScraper(config) as scraper:
    result = scraper.scrape(query="laptop")

A scraping-API service

Services like ScraperAPI, ScrapeOps, and ScrapingBee handle proxies and anti-bot challenges for you. PyScrappy routes requests through them transparently:

from pyscrappy import ScraperConfig, AmazonScraper
 
config = ScraperConfig(scraper_api={
    "provider": "scraperapi",   # or "scrapeops", "scrapingbee"
    "api_key": "YOUR_KEY",
    "render_js": True,          # optional
})
 
with AmazonScraper(config) as scraper:
    result = scraper.scrape(query="laptop")

This is the reliable way to use the scrapers marked needs proxy in the scraper catalog, for example Instagram, Twitter/X, and Spotify.

Start without a proxy. Only add one when a specific scraper reports it's being blocked. Most of PyScrappy's scrapers use clean public APIs and don't need one.