Adaptive selectors

A hard-coded CSS selector silently breaks the day a site changes its markup. Adaptive selectors survive that: save a fingerprint of the element the first time, and if the selector later matches nothing, relocate it by structural and textual similarity instead of returning empty.

from pyscrappy import Selector
 
# First run: match normally and remember this element under an id.
page = Selector(html_v1, url="https://shop.example.com")
price = page.css(".price", auto_save=True, adaptive_id="price").get()
 
# Later, after a redesign renamed ".price" — heal instead of breaking:
page = Selector(html_v2, url="https://shop.example.com")
result = page.css(".price", adaptive=True, adaptive_id="price")
print(result.get(), "→ confidence:", result.adaptive_confidence)

• auto_save=True fingerprints the first matched element under adaptive_id (defaults to the selector string).
• adaptive=True — if the selector matches nothing, relocate the saved element.
• threshold (default 55) is the minimum confidence, 0–100, to accept a match.
• Adaptive is fully opt-in: without adaptive=True, a broken selector still returns empty, exactly as before.

How relocation decides

Relocation scores every candidate against the saved fingerprint. It's designed to stay right where a naive similarity match goes wrong:

• Weighted signals, not a flat average. A stable id / data-* hook counts far more than a sibling-tag list, so weak signals can't outvote strong ones.
• Anchor-relative. The fingerprint records the nearest stable ancestor (an id'd / data-* container) and depth, so it survives layout reshuffles that move absolute positions.
• Volatility-aware text. Prices, dates, and counts are down-weighted, so healing stays reliable on exactly the fields that change most between scrapes.
• Confidence-scored. SelectorList.adaptive_confidence (0–100) tells you how sure the relocation was, and the runner-up gap distinguishes a decisive match from an ambiguous one.

Where fingerprints live

Fingerprints persist in a small JSON store at ~/.pyscrappy/adaptive.json by default (override the directory with $PYSCRAPPY_HOME). They're namespaced by site (from the url= you pass), so the same adaptive_id on two different sites never collides.