Configuration | PyScrappy

Control timeouts, retries, rate limits, proxies, and caching with ScraperConfig.

Every scraper accepts an optional ScraperConfig. It controls timeouts, retries, rate limiting, proxies, browser behavior, and caching.

from pyscrappy import ScraperConfig, GenericScraper
 
config = ScraperConfig(
    timeout=20.0,            # request timeout in seconds
    max_retries=3,           # retry failed requests
    rate_limit=2.0,          # seconds between requests per domain
    proxy="http://...",      # proxy URL, or a list to rotate through
    scraper_api=None,        # route via a scraping-API service (see Proxies)
    headless=True,           # browser runs headless
    render_js="auto",        # auto-detect if JS rendering is needed
    cache_ttl=0,             # response cache TTL in seconds (0 = disabled)
)
 
with GenericScraper(config) as gs:
    result = gs.scrape(url="https://example.com")

Options

Option	Type	Default	Description
`timeout`	`float`	`20.0`	Per-request timeout, in seconds.
`max_retries`	`int`	`3`	Retries on failure, with exponential backoff.
`rate_limit`	`float`	-	Minimum seconds between requests per domain.
`proxy`	`str \| list[str]`	`None`	A proxy URL, or a list rotated one-per-request.
`scraper_api`	`dict`	`None`	Route through ScraperAPI / ScrapeOps / ScrapingBee.
`headless`	`bool`	`True`	Whether the browser backend runs headless.
`render_js`	`"auto" \| bool`	`"auto"`	Whether to render JavaScript with a browser.
`cache_ttl`	`float`	`0`	Cache successful GETs for this many seconds (`0` disables).

Applying config to any scraper

The same config object works for every scraper, so pass it to the constructor:

from pyscrappy import ScraperConfig, AmazonScraper
 
config = ScraperConfig(timeout=30.0, max_retries=5, rate_limit=1.0)
 
with AmazonScraper(config) as scraper:
    result = scraper.scrape(query="laptop")

See also: Proxies & blocked sites, Concurrency, and Caching.