Concurrency | PyScrappy

Scraping is I/O-bound, so running several scrapes at once parallelizes the network waits. PyScrappy provides two helpers. Both preserve input order.

scrape_many: run one scraper over many inputs.
scrape_all: run a mix of scrapers together.

from pyscrappy import (
    scrape_many, scrape_all,
    AmazonScraper, WikipediaScraper, NewsScraper,
)
 
# One scraper, many queries, concurrently:
results = scrape_many(AmazonScraper, [{"query": "laptop"}, {"query": "phone"}])
 
# Different scrapers at once:
results = scrape_all([
    lambda: WikipediaScraper().scrape(query="Python"),
    lambda: NewsScraper().scrape(
        feed_url="https://rss.nytimes.com/services/xml/rss/nyt/World.xml"
    ),
])

Each entry in the returned list is a ScrapeResult, in the same order as the inputs.

Concurrency is implemented with a thread pool, which suits I/O-bound scrapers well. Combine it with rate_limit in your ScraperConfig to stay polite to each domain.