Guides, Tutorials & Insights
Learn about residential proxies, web scraping, data collection strategies, and industry best practices from the Shifter team.
Character Encodings in Scraped Data: Detecting Charsets and Fixing Mojibake
We fetched 193 top homepages in six countries. A common Python default garbled the text of 37 of them. How to decode scraped pages the way browsers do.
Chris Collins · October 1, 2026 Keep the Raw Response: Archiving Scraped Pages in WARC for Replay
Parsers improve and extractors break, but a page you did not keep cannot be parsed again. How to store raw responses in WARC, with tested code and real sizes.
Matt Brown · October 1, 2026 Normalising Prices, Numbers and Dates Across Locales
1.234,56 € and $1,234.56 are both prices, and 03/04/2026 is two different dates. How to parse scraped values correctly by locale, with tested code.
James Meadow · September 30, 2026 Finding the API Behind the Page: Collecting From JSON Endpoints Instead of HTML
Many pages load their data as JSON. How to find the endpoint that carries it, why it is often tied to the page's session, and how to collect it responsibly.
Chris Collins · September 30, 2026 Introducing IP Info: A Free IP Geolocation API, No API Key Needed
IP Info is Shifter's free IP geolocation and ASN API. It runs on the same database behind our proxy pools, needs no signup or API key, and is ready for AI agents.
James Meadow · September 29, 2026 Schema Drift in Scraped Data: Catching Broken Extractors Before Your Users Do
Scrapers rarely crash when a site changes. They keep running and return subtly wrong data. How data contracts and batch profiling catch drift early.
Matt Brown · September 29, 2026 Sitemaps as a Discovery Source: Finding Every URL Without Crawling the Whole Site
XML sitemaps list a site's URLs, and sometimes when they changed. How to read them properly, and why you should never trust lastmod without checking it.
James Meadow · September 29, 2026 LLM Extraction vs Selectors: When to Let a Model Read the Page
We had a large language model extract fields from 18 live news pages and scored it against the pages' own structured data. Accuracy, cost, and when to use it.
Chris Collins · September 28, 2026 Deduplicating Scraped Data: Entity Resolution for Products, Companies and Listings
The same product, company or listing shows up many times across sources and runs. How to normalise identifiers, match safely at scale and keep one clean record.
Matt Brown · September 28, 2026