Guides
Caching
Cache successful responses in memory to skip the network and rate limiter.
Set cache_ttl to a positive number of seconds to cache successful GET responses.
Repeated requests for the same URL (and query params) within the TTL are served from
cache, skipping both the network and the rate limiter. Caching is disabled by
default (cache_ttl=0).
from pyscrappy import WikipediaScraper, ScraperConfig
config = ScraperConfig(cache_ttl=300) # cache for 5 minutes
with WikipediaScraper(config) as ws:
ws.scrape(query="Python") # fetched over the network
ws.scrape(query="Python") # served from cacheHow it works
- The cache is in memory and shared across scraper instances in the same process, so it also speeds up repeated calls through the MCP server.
- It is cleared when the process exits.
- Clear it manually with
HttpClient.clear_cache().
from pyscrappy.core.http import HttpClient
HttpClient.clear_cache() # empty the shared cache