Writing a plugin

PyScrappy is extensible. You can register your own scrapers in-process, or ship them as standalone pyscrappy-<name> packages that PyScrappy discovers automatically. A registered scraper works everywhere a built-in does: the Python API, the MCP server, and the pyscrappy chat agent — with no change to PyScrappy core.

The contract

A scraper subclasses BaseScraper, sets a name, and implements scrape() returning a ScrapeResult (whose data is a list of dicts). BaseScraper gives you fetching, parsing, retries, rate limiting, proxies, and caching, all driven by ScraperConfig.

from pyscrappy import BaseScraper, register_scraper
from pyscrappy.core.models import ScrapeResult, ScrapeMetadata
 
@register_scraper("reddit")
class RedditScraper(BaseScraper):
    def scrape(self, subreddit: str, **kwargs) -> ScrapeResult:
        soup = self.fetch_and_parse(f"https://old.reddit.com/r/{subreddit}/")
        posts = [
            {"title": a.get_text(strip=True), "url": a["href"]}
            for a in soup.select("a.title")
        ]
        return ScrapeResult(data=posts, metadata=ScrapeMetadata(scraper="reddit"))

Use it immediately:

from pyscrappy import get_scraper
 
with get_scraper("reddit")() as s:
    result = s.scrape(subreddit="python")
    print(result.to_markdown())

Shipping it as a package

To let others pip install your scraper, publish a pyscrappy-<name> package that advertises an entry point in the pyscrappy.scrapers group:

# pyproject.toml of your pyscrappy-reddit package
[project.entry-points."pyscrappy.scrapers"]
reddit = "pyscrappy_reddit:RedditScraper"

That's the whole integration. Once the package is installed, PyScrappy discovers it lazily — list_scrapers() includes it, get_scraper("reddit") resolves it, and it needs no @register_scraper decorator (the entry point does the registration).

Plugins are agent-ready automatically

Any registered scraper — built-in or plugin — is exposed to AI agents without extra work. The MCP server ships two generic tools:

• list_available_scrapers — returns every scraper name, including plugins.
• scrape_with(name, args) — runs any scraper by name with arbitrary arguments.

So the moment a user installs your pyscrappy-reddit package, an agent connected to PyScrappy's MCP server can discover and call it:

"Use pyscrappy to get the top posts from r/python."

The agent calls list_available_scrapers, sees reddit, and runs scrape_with(name="reddit", args={"subreddit": "python"}). No MCP glue required from you.

Starting point

The repo includes a complete, copyable plugin template: a renamable package with the entry point wired up and a passing test. Copy it, rename it, implement scrape(), pip install -e ., and you're discoverable.

API reference