Guides, Tutorials & Insights
Learn about residential proxies, web scraping, data collection strategies, and industry best practices from the Shifter team.
Character Encodings in Scraped Data: Detecting Charsets and Fixing Mojibake
We fetched 193 top homepages in six countries. A common Python default garbled the text of 37 of them. How to decode scraped pages the way browsers do.
Chris Collins · October 1, 2026 Keep the Raw Response: Archiving Scraped Pages in WARC for Replay
Parsers improve and extractors break, but a page you did not keep cannot be parsed again. How to store raw responses in WARC, with tested code and real sizes.
Matt Brown · October 1, 2026 Finding the API Behind the Page: Collecting From JSON Endpoints Instead of HTML
Many pages load their data as JSON. How to find the endpoint that carries it, why it is often tied to the page's session, and how to collect it responsibly.
Chris Collins · September 30, 2026 Sitemaps as a Discovery Source: Finding Every URL Without Crawling the Whole Site
XML sitemaps list a site's URLs, and sometimes when they changed. How to read them properly, and why you should never trust lastmod without checking it.
James Meadow · September 29, 2026 LLM Extraction vs Selectors: When to Let a Model Read the Page
We had a large language model extract fields from 18 live news pages and scored it against the pages' own structured data. Accuracy, cost, and when to use it.
Chris Collins · September 28, 2026 Change Detection at Scale: Diffing Web Pages Without Drowning in Noise
Pages change on every load: ads, timestamps, tokens. How to tell real changes from noise with normalisation, field-level diffs and simhash, with tested code.
Chris Collins · September 27, 2026 Stop Parsing HTML: Extracting Structured Data From JSON-LD and Embedded Data
Most pages already carry machine-readable data. How to extract JSON-LD and embedded JSON instead of brittle selectors, and what to do when it is incomplete.
Matt Brown · September 26, 2026 Building a Target Health Score: Knowing a Site Is Turning Against You
Sites rarely block a crawler in one step. They challenge, soft-block and slow down first. How to turn those signals into one score per site, and act on it.
Matt Brown · September 22, 2026 Cost-Aware Crawl Scheduling: Spending Your Budget Where It Changes
Most recrawl budget is spent confirming nothing changed. How to schedule by expected value per cost, and why the pages that change fastest are not the best buy.
James Meadow · September 22, 2026