Quickstart | PyScrappy

Every scraper follows the same shape: construct it (optionally with a ScraperConfig), call .scrape(...), and read the typed ScrapeResult.

Scrape any URL

from pyscrappy import scrape
 
result = scrape("https://en.wikipedia.org/wiki/Web_scraping")
print(result.data[0]["metadata"]["title"])
print(result.data[0]["text"]["word_count"])

This is a fast static fetch and does not run JavaScript. If a page builds its content client-side (so the scrape comes back nearly empty), pass render_js=True to scrape it in a headless browser instead:

result = scrape("https://github.com/torvalds", render_js=True)

render_js needs the browser extra (pip install 'pyscrappy[browser]' and playwright install chromium). Alternatively, scrape the static data endpoint the page fetches, e.g. https://github.com/users/<name>/contributions. See Proxies & blocked sites for sites that also need a proxy.

Custom CSS selectors

Pass selectors to extract exactly the fields you want:

from pyscrappy import GenericScraper
 
with GenericScraper() as gs:
    result = gs.scrape(
        url="https://news.ycombinator.com",
        selectors={"title": ".titleline a", "score": ".score"},
    )
    for item in result.data:
        print(item["title"], item.get("score", ""))

Use a built-in scraper

from pyscrappy import WikipediaScraper
 
with WikipediaScraper() as ws:
    result = ws.scrape(query="Python (programming language)", mode="summary")
    print(result.data[0]["text"])

Get data as a DataFrame

Any result can become a pandas DataFrame (requires the dataframe extra):

from pyscrappy import StockScraper
 
with StockScraper() as ss:
    result = ss.scrape(symbol="AAPL", mode="history", period="1mo")
    df = result.to_dataframe()
    print(df.head())

The ScrapeResult object

Every scraper returns a ScrapeResult:

result = scrape("https://example.com")
 
result.data          # list of dicts: the scraped items
result.count         # number of items
result.source_urls   # URLs that were fetched
result.errors        # non-fatal problems (each has .url and .message)
 
result.to_dataframe()  # -> pandas.DataFrame  (needs the dataframe extra)
result.to_json()       # -> JSON string

Because the envelope is always the same, you can swap scrapers without changing how you handle the output.

Next steps

Configuration: timeouts, retries, rate limits, proxies.
Scrapers: the full catalog.
MCP server: call PyScrappy from an AI agent.