Guides, Tutorials & Insights
Learn about residential proxies, web scraping, data collection strategies, and industry best practices from the Shifter team.
Cost per Clean Record: The Scraping Metric Finance Actually Understands
Gigabytes and request counts do not tell finance what data costs. How to measure cost per clean record and per useful change, and which levers move it most.
James Meadow · September 27, 2026 Change Detection at Scale: Diffing Web Pages Without Drowning in Noise
Pages change on every load: ads, timestamps, tokens. How to tell real changes from noise with normalisation, field-level diffs and simhash, with tested code.
Chris Collins · September 27, 2026 Stop Parsing HTML: Extracting Structured Data From JSON-LD and Embedded Data
Most pages already carry machine-readable data. How to extract JSON-LD and embedded JSON instead of brittle selectors, and what to do when it is incomplete.
Matt Brown · September 26, 2026 robots.txt, AI Opt-Outs and Reservation Signals: What Web Data Collectors Should Honour in 2026
robots.txt now carries legal weight in the EU, and new AI signals are spreading. What each one means, what AI crawlers honour and how data teams should respond.
James Meadow · September 26, 2026 How Brands Build Court-Ready Evidence From Online Counterfeit Listings
Finding a counterfeit listing is easy. Proving it later is not. How to capture, hash, timestamp and preserve listing evidence that holds up in court.
Chris Collins · September 25, 2026 Verifying ESG Claims With Public Web Evidence
New EU rules on green claims apply from 27 September 2026. A practical method for checking sustainability claims against public evidence, and tracking changes.
Matt Brown · September 25, 2026 Monitoring Your Suppliers' Public Footprint: A Practical Collection Method
Supplier risk usually shows up in public before it reaches an audit. A step-by-step method for monitoring sanctions lists, registries, websites and local news.
James Meadow · September 25, 2026 Do Airlines and Hotels Quote You Differently by Exit Country? We Checked
We loaded Ryanair fares and Booking.com hotel prices from six countries through residential IPs. What changed, what did not and what we could not reach.
Chris Collins · September 24, 2026 Weekday, Weekend, Holiday: How Block Rates Move With the Calendar
Nobody publishes scraper block rates by day of week. What bot, holiday and peak-event data shows about why they move, and how to measure your own calendar.
Matt Brown · September 24, 2026