Guides, Tutorials & Insights
Learn about residential proxies, web scraping, data collection strategies, and industry best practices from the Shifter team.
Testing Scrapers: Fixtures, Replay and Regression Suites
Scrapers break when sites change, and tests that hit the live site break with them. How to test a scraper with recorded fixtures, replay and daily canaries.
Chris Collins · October 9, 2026 Your Scraper Is Slow and It Probably Isn't the Proxy
We sped up a scraper running through residential proxies 7.5 times without touching the proxy. How to find where the time really goes, and fix it.
Matt Brown · October 8, 2026 Payload Efficiency: What Share of a Scraped Page Is Boilerplate You Paid For
We measured 278 content pages from the top 1,000 sites. The text you want is about 1% of the HTML and a tiny share of what a browser downloads.
Elena Petrova · October 8, 2026 Who Blocks the AI Crawlers? GPTBot, ClaudeBot and Friends Across the Top 10,000 Sites
We read the robots.txt of the top 10,000 domains. One site in five blocks at least one AI crawler, while almost all still welcome search engines.
Chris Collins · October 6, 2026 The Cost of a Stale User-Agent: Measuring Browser Version Drift
Scrapers often hardcode a browser version and forget it. We requested 397 top sites claiming Chrome versions up to 6.5 years old. Refusals rose every year.
Chris Collins · October 5, 2026 Is Your Target Blockable? An Anti-Bot Stack Lookup
Before building a scraper, find out what protects the target. We checked the top 1,000 domains: which anti-bot vendors they use and how a plain client fared.
Chris Collins · October 3, 2026 Proxy Selection as a Multi-Armed Bandit Problem
Which proxy configuration should the next request use? Treat it as a bandit problem. In our simulation, adaptive selection came within 2% of perfect hindsight.
James Meadow · October 2, 2026 Designing Idempotent Scrapers: Restarting Without Duplicating
Scrapers crash. We killed one ten times mid-crawl: a naive version stored up to 4.3 times the rows, an idempotent one exactly 500. How to build the second.
Chris Collins · October 2, 2026