Guides, Tutorials & Insights
Learn about residential proxies, web scraping, data collection strategies, and industry best practices from the Shifter team.
Subnet Contagion: When One Blocked IP Poisons Its Neighbours
Blocklists and reputation systems often judge an IP by its neighbours. What research shows about range-level blocking, its collateral damage and how to spot it.
James Meadow · September 23, 2026 The Silent Failure Rate: What Share of Successful Requests Return the Wrong Content
An HTTP 200 does not mean you got the page. What research shows about block pages, soft 404s and challenges hiding behind success, and how to measure your own.
Chris Collins · September 23, 2026 Building a Target Health Score: Knowing a Site Is Turning Against You
Sites rarely block a crawler in one step. They challenge, soft-block and slow down first. How to turn those signals into one score per site, and act on it.
Matt Brown · September 22, 2026 Cost-Aware Crawl Scheduling: Spending Your Budget Where It Changes
Most recrawl budget is spent confirming nothing changed. How to schedule by expected value per cost, and why the pages that change fastest are not the best buy.
James Meadow · September 22, 2026 Backpressure and Flow Control in Distributed Crawlers
A crawler is a pipeline, and its fastest stage will flood its slowest. How bounded queues, pull-based work and adaptive per-host limits keep a crawler stable.
Chris Collins · September 22, 2026 Running Scrapers in CI/CD: Proxy Credentials, Secrets and Test Runs
How to run scrapers in CI without leaking proxy credentials or burning bandwidth on every push: secret storage, test tiers, sessions, quota checks and rotation.
Chris Collins · September 21, 2026 Do SEO Tools Report Real Search Volume? Three Vendors Against SERP Reality
Keyword Planner, Semrush and Ahrefs each estimate search volume differently. What independent tests found, and why the live SERP matters more than the number.
Matt Brown · September 21, 2026 Which Countries Get Geo-Blocked Most, and by Whom
Sanctioned countries top every measurement of geo-blocking, but they are not the whole story. What the research shows about who blocks whom, and how.
James Meadow · September 21, 2026 Ad Load by Market: How Much of a Page Is Advertising, Country by Country
Nobody has published ad bytes as a share of page bytes by country. Here is what the data does show, and why Chrome just made the real answer possible.
Matt Brown · September 20, 2026