Data & research scrapers

Wikipedia, stocks, news, IMDB, images, weather, crypto, currency, and dictionary.

These scrapers pull from clean public sources (many via official APIs), so they work without a proxy.

Wikipedia

from pyscrappy import WikipediaScraper
 
with WikipediaScraper() as ws:
    result = ws.scrape(query="Python (programming language)", mode="summary")
    print(result.data[0]["text"])

mode can be "full", "summary", "paragraphs", or "headers".

Stock data (Yahoo Finance)

from pyscrappy import StockScraper
 
with StockScraper() as ss:
    result = ss.scrape(symbol="AAPL", mode="history", period="1mo")
    df = result.to_dataframe()
    print(df.head())

mode can be "quote", "history", or "profile"; period accepts values like "1mo", "1y".

News (RSS feeds)

from pyscrappy import NewsScraper
 
with NewsScraper() as ns:
    result = ns.scrape(
        feed_url="https://rss.nytimes.com/services/xml/rss/nyt/World.xml"
    )
    for article in result.data[:5]:
        print(article["title"])

You can also pass a site_url (its feed is auto-discovered) or an article_url (full-text extraction).

IMDB (via OMDb API)

IMDB's own pages are protected by an anti-bot challenge, so PyScrappy fetches IMDB data through the free OMDb API. Set an OMDb API key in the OMDB_API_KEY environment variable (or pass api_key=...).

from pyscrappy import IMDBScraper
 
with IMDBScraper() as scraper:          # reads OMDB_API_KEY from the environment
    result = scraper.scrape(query="inception")      # search by title
    result = scraper.scrape(query="tt1375666")      # or look up an IMDB id
    df = result.to_dataframe()
    print(df[["title", "year", "rating", "genre"]])

Image search

from pyscrappy import ImageSearchScraper
 
with ImageSearchScraper() as iss:
    result = iss.scrape(query="golden retriever", max_images=10, download_to="./dogs")

Weather, crypto, currency, dictionary

These use key-free public APIs (Open-Meteo, CoinGecko, exchange rates, Free Dictionary):

from pyscrappy import WeatherScraper, CryptoScraper, CurrencyScraper, DictionaryScraper
 
WeatherScraper().scrape(location="Tokyo, Japan")
CryptoScraper().scrape(query="bitcoin, ethereum")
CurrencyScraper().scrape(base="USD", to="EUR,GBP", amount=100)
DictionaryScraper().scrape(word="serendipity")