Guides, Tutorials & Insights
Learn about residential proxies, web scraping, data collection strategies, and industry best practices from the Shifter team.
Deduplicating Scraped Data: Entity Resolution for Products, Companies and Listings
The same product, company or listing shows up many times across sources and runs. How to normalise identifiers, match safely at scale and keep one clean record.
Matt Brown · September 28, 2026 Cost per Clean Record: The Scraping Metric Finance Actually Understands
Gigabytes and request counts do not tell finance what data costs. How to measure cost per clean record and per useful change, and which levers move it most.
James Meadow · September 27, 2026 robots.txt, AI Opt-Outs and Reservation Signals: What Web Data Collectors Should Honour in 2026
robots.txt now carries legal weight in the EU, and new AI signals are spreading. What each one means, what AI crawlers honour and how data teams should respond.
James Meadow · September 26, 2026 How Brands Build Court-Ready Evidence From Online Counterfeit Listings
Finding a counterfeit listing is easy. Proving it later is not. How to capture, hash, timestamp and preserve listing evidence that holds up in court.
Chris Collins · September 25, 2026 Verifying ESG Claims With Public Web Evidence
New EU rules on green claims apply from 27 September 2026. A practical method for checking sustainability claims against public evidence, and tracking changes.
Matt Brown · September 25, 2026 Monitoring Your Suppliers' Public Footprint: A Practical Collection Method
Supplier risk usually shows up in public before it reaches an audit. A step-by-step method for monitoring sanctions lists, registries, websites and local news.
James Meadow · September 25, 2026 Do Airlines and Hotels Quote You Differently by Exit Country? We Checked
We loaded Ryanair fares and Booking.com hotel prices from six countries through residential IPs. What changed, what did not and what we could not reach.
Chris Collins · September 24, 2026 Weekday, Weekend, Holiday: How Block Rates Move With the Calendar
Nobody publishes scraper block rates by day of week. What bot, holiday and peak-event data shows about why they move, and how to measure your own calendar.
Matt Brown · September 24, 2026 Subnet Contagion: When One Blocked IP Poisons Its Neighbours
Blocklists and reputation systems often judge an IP by its neighbours. What research shows about range-level blocking, its collateral damage and how to spot it.
James Meadow · September 23, 2026