Most scrapers are tested the same way: someone runs them against the live site, looks at the output, and decides it looks right. That works until the day it does not. The site changes a class name, the scraper returns empty fields, and nobody notices until a report downstream comes out wrong.
The difficulty is that a scraper depends on something you do not control. Tests that call the live site are slow, flaky and change their answers without warning, so teams stop running them. Tests that never touch real pages pass forever while production breaks.
The way out is to split the problem in two. Test your code against recorded pages that never change, and test the site separately with a small daily check that tells you when it has. This guide builds both, with code you can adapt: recorded fixtures, a replay session that runs the whole crawler offline, golden-file tests for the parser, and a canary for the live site.
Three things that can break
When a scraper produces bad data, the cause is almost always one of three things, and each needs a different kind of test:
| What broke | Example | Test that catches it |
|---|---|---|
| Your parser | A refactor turns “1,099.00” into 1.099 | Golden-file tests on recorded pages |
| Your crawl logic | Pagination stops after page one; a 503 is treated as an empty page | Replay tests that run the crawler offline |
| The site | A class name changes, a field moves, a block page appears | A daily canary against the live site |
The first two are bugs in your code and should be caught before you deploy, in tests that run in seconds without a network. The third is not a bug at all; it is the world changing, and it needs monitoring rather than a unit test.
Keep fetching and parsing apart
Testing starts with structure. If the function that parses a page also fetches it, every test has to deal with the network. Keep parsing a pure function: HTML in, records out. The crawler then becomes a thin loop that fetches pages and hands them to the parser.
Here is the scraper we will test, for a listing site with pagination:
from urllib.parse import urljoin
from lxml import html as lxml_html
def parse_listing(page_html):
"""Products on one listing page. Pure function: HTML in, records out."""
doc = lxml_html.fromstring(page_html)
items = []
for card in doc.cssselect("li.product"):
price = card.cssselect(".price")[0].text_content().strip()
items.append({
"sku": card.get("data-sku"),
"name": card.cssselect("h2")[0].text_content().strip(),
"price": float(price.replace("$", "").replace(",", "")),
"in_stock": not card.cssselect(".sold-out"),
})
return items
def next_page(page_html, base_url):
links = lxml_html.fromstring(page_html).cssselect("a[rel=next]")
return urljoin(base_url, links[0].get("href")) if links else None
def crawl(session, start_url, max_pages=50):
"""Follow pagination; return the records and the URLs that failed."""
records, failed, url = [], [], start_url
for _ in range(max_pages):
response = session.get(url, timeout=30)
if response.status_code != 200:
failed.append((url, response.status_code))
break
records.extend(parse_listing(response.text))
url = next_page(response.text, response.url)
if url is None:
break
return records, failedcrawl() takes the session as an argument instead of creating one. That single choice is what makes it testable: in production you pass a requests.Session, in tests you pass something that serves recorded pages.
Record fixtures once
A fixture is a real response, saved to disk with enough metadata to replay it exactly: the URL, the final URL after redirects, the status code and the encoding. Record the body as raw bytes, not decoded text, so encoding problems are reproduced rather than hidden.
import hashlib
import json
from pathlib import Path
def _name(url):
return hashlib.sha256(url.encode()).hexdigest()[:16]
def record(session, url, directory):
"""Fetch a page once and save everything a test needs to replay it."""
response = session.get(url, timeout=30)
directory = Path(directory)
directory.mkdir(parents=True, exist_ok=True)
name = _name(url)
(directory / f"{name}.html").write_bytes(response.content)
(directory / f"{name}.json").write_text(json.dumps({
"url": url,
"final_url": response.url,
"status": response.status_code,
"encoding": response.encoding,
"headers": {"Content-Type": response.headers.get("Content-Type", "")},
}, indent=2))
return name
class Replayed:
def __init__(self, meta, body):
self.url = meta["final_url"]
self.status_code = meta["status"]
self.headers = meta["headers"]
self.content = body
self.text = body.decode(meta["encoding"] or "utf-8", errors="replace")
class ReplaySession:
"""Stands in for requests.Session: serves recorded pages, never the network."""
def __init__(self, directory):
self.directory = Path(directory)
self.requested = []
def get(self, url, **kwargs):
self.requested.append(url)
name = _name(url)
meta_path = self.directory / f"{name}.json"
if not meta_path.exists():
raise LookupError(f"no fixture recorded for {url}")
meta = json.loads(meta_path.read_text())
return Replayed(meta, (self.directory / f"{name}.html").read_bytes())ReplaySession has the same get() method as requests.Session, so crawl() cannot tell the difference. It raises an error for any URL that was not recorded, which matters: a test that silently fell back to the network would be fast on your laptop and slow and flaky in CI.
Record more than happy pages. The cases that break scrapers in production are the ones worth keeping: the last page of pagination, an empty search result, a sold-out product, a 503, a block page. Each becomes a permanent test case. If you archive raw responses in production, as our guide to archiving scraped pages in WARC describes, those archives are a ready source of fixtures from real failures.
Validate what any correct crawl must satisfy
Golden files check exact output. Validation checks rules that must hold however the site’s content changes: every record has an ID, IDs are unique, names are not empty, prices are plausible numbers, and the crawl returned roughly as many records as expected.
def validate(records, min_records=1):
"""Checks that hold for any correct crawl, whatever the site shows today."""
problems = []
if len(records) < min_records:
problems.append(f"only {len(records)} records, expected at least {min_records}")
seen = set()
for i, r in enumerate(records):
if not r.get("sku"):
problems.append(f"record {i}: missing sku")
elif r["sku"] in seen:
problems.append(f"record {i}: duplicate sku {r['sku']}")
seen.add(r.get("sku"))
if not r.get("name"):
problems.append(f"record {i}: empty name")
if not isinstance(r.get("price"), float) or not 0 < r["price"] < 100_000:
problems.append(f"record {i}: implausible price {r.get('price')!r}")
return problems
def canary(session, start_url, crawl, min_records):
"""Run the real crawl against the live site and list anything wrong.
Schedule it daily: an empty list means the scraper still works today.
"""
records, failed = crawl(session, start_url)
return validate(records, min_records) + [f"fetch failed: {url} ({status})" for url, status in failed]The same validate() runs in three places: in the test suite, in the daily canary, and, ideally, on every production run before data is written. Our guide to schema drift in scraped data goes further into field-level checks over time.
The test suite
With fixtures and validation in place, the tests are short. They use only the standard library:
import json
import os
import unittest
from pathlib import Path
from fixtures import ReplaySession
from shop import crawl, parse_listing
from validate import canary, validate
FIXTURES = Path(__file__).parent / "fixtures"
GOLDEN = Path(__file__).parent / "golden"
UPDATE = os.environ.get("UPDATE_GOLDEN") == "1"
class ParserTests(unittest.TestCase):
def test_every_fixture_matches_its_golden_file(self):
for page in sorted(FIXTURES.glob("*.html")):
meta = json.loads(page.with_suffix(".json").read_text())
if meta["status"] != 200:
continue
with self.subTest(url=meta["url"]):
parsed = parse_listing(page.read_text())
golden = GOLDEN / f"{page.stem}.json"
if UPDATE:
golden.write_text(json.dumps(parsed, indent=2))
self.assertEqual(parsed, json.loads(golden.read_text()))
class CrawlTests(unittest.TestCase):
def test_crawl_follows_pagination_offline(self):
session = ReplaySession(FIXTURES)
records, failed = crawl(session, "https://shop.example/laptops?page=1")
self.assertEqual(len(session.requested), 2)
self.assertEqual(failed, [])
self.assertEqual(validate(records, min_records=4), [])
def test_crawl_reports_a_failed_page(self):
session = ReplaySession(FIXTURES)
records, failed = crawl(session, "https://shop.example/monitors?page=1")
self.assertEqual(failed, [("https://shop.example/monitors?page=2", 503)])
self.assertEqual(len(records), 2)
class ValidationTests(unittest.TestCase):
def test_catches_broken_records(self):
bad = [{"sku": "A1", "name": "", "price": 0.0}, {"sku": "A1", "name": "x", "price": "12"}]
self.assertEqual(validate(bad), [
"record 0: empty name",
"record 0: implausible price 0.0",
"record 1: duplicate sku A1",
"record 1: implausible price '12'",
])
def test_canary_flags_a_partial_crawl(self):
problems = canary(ReplaySession(FIXTURES), "https://shop.example/monitors?page=1", crawl, min_records=4)
self.assertEqual(problems, [
"only 2 records, expected at least 4",
"fetch failed: https://shop.example/monitors?page=2 (503)",
])
if __name__ == "__main__":
unittest.main()- The golden-file test parses every recorded page and compares the result with a stored JSON file. The first time, run it with
UPDATE_GOLDEN=1to write the golden files, then review them by hand: they are the specification of what the parser should produce. After that, any change in parser output fails the test until someone looks at it and decides whether it is a fix or a bug. - The replay tests run the real
crawl()offline. One checks that pagination is followed to the end; the other checks that a 503 on page two is reported rather than silently treated as the end of the results. - The validation tests check that the rules actually catch broken records, and that the canary reports a partial crawl.
Running the suite against fixtures recorded from a small test shop:
test_crawl_follows_pagination_offline (test_shop.CrawlTests) ... ok
test_crawl_reports_a_failed_page (test_shop.CrawlTests) ... ok
test_every_fixture_matches_its_golden_file (test_shop.ParserTests) ... ok
test_canary_flags_a_partial_crawl (test_shop.ValidationTests) ... ok
test_catches_broken_records (test_shop.ValidationTests) ... ok
----------------------------------------------------------------------
Ran 5 tests in 0.019s
OKFive tests in a fraction of a second, with no network. That speed is the point: tests this fast run on every commit. Our guide to running scrapers in CI/CD covers the pipeline around them, including why fixture tests need no proxy credentials at all.
The canary: testing the site, not your code
Fixtures freeze the site as it was. To find out when it changes, run the real crawl against the live site on a schedule and validate the result. That is the canary() function above: it returns an empty list when everything is fine and a list of specific problems when it is not.
Run it once a day, or before each large crawl, with a small start URL and a realistic minimum record count. When it fails:
- Record the page that broke as a new fixture.
- Watch the golden-file or replay test fail on it, which confirms the test reproduces the problem.
- Fix the parser, review the new golden output, and commit the fixture, the fix and the golden file together.
The site change becomes a permanent regression test, so the same break can never come back unnoticed. That loop, canary to fixture to fix, is what turns a fragile scraper into a maintained one.
A canary also tells you about failures that are not about parsing at all: a block page that returns status 200, a region redirect, or a consent wall in place of content. Our study of the silent failure rate shows how common responses that look successful but carry the wrong content are. For ongoing monitoring of a whole pipeline, see how to monitor a web scraping pipeline.
Practical rules
- Never call the network from unit tests. Make missing fixtures an error, as
ReplaySessiondoes. - Keep fixtures small and real. One page per case is enough. Trim nothing from the HTML: the noise you would cut is often what breaks a parser.
- Review golden files like code. An unreviewed golden file only proves the parser does what it did yesterday.
- Name the case, not the date. A fixture called “sold-out product” is useful in a year; one called “Tuesday’s page” is not.
- Refresh fixtures deliberately. When the site changes, add new fixtures alongside the old ones rather than replacing them, unless the old layout can never come back.
- Keep secrets out of fixtures. Recorded pages can contain session tokens or personal data. Strip cookies and check the bodies before committing them.
FAQ
How do you test a web scraper without hitting the live site?
Record real responses once as fixtures, then replace the HTTP session with one that serves those recordings. The parser and crawler run exactly as in production, but offline, in milliseconds and with the same answer every time.
What is a golden file test?
A test that compares a function’s output with a stored, reviewed copy of the expected output. For a scraper, the golden file is the list of records a recorded page should produce. Any change in the output fails the test until someone accepts or fixes it.
How do I know when a website change breaks my scraper?
Run a canary: a small real crawl against the live site, on a schedule, with validation rules that any correct result must satisfy. When it reports problems, record the new page as a fixture, fix the parser and keep the fixture as a regression test.
Should scraper tests use proxies?
Unit and replay tests should not use the network at all, so they need no proxies or credentials. The canary runs against the live site and should use the same proxy setup as production, so that it sees what production sees.
The bottom line
Test your code and test the site separately. Recorded fixtures and a replay session let you test the parser and the crawler offline, quickly and deterministically, on every commit. A daily canary with validation rules tells you when the site changes. Turn every break the canary finds into a fixture, and the scraper gets more reliable each time the site changes, not less.