Guides, Tutorials & Insights
Learn about residential proxies, web scraping, data collection strategies, and industry best practices from the Shifter team.
Backpressure and Flow Control in Distributed Crawlers
A crawler is a pipeline, and its fastest stage will flood its slowest. How bounded queues, pull-based work and adaptive per-host limits keep a crawler stable.
Chris Collins · September 22, 2026 Running Scrapers in CI/CD: Proxy Credentials, Secrets and Test Runs
How to run scrapers in CI without leaking proxy credentials or burning bandwidth on every push: secret storage, test tiers, sessions, quota checks and rotation.
Chris Collins · September 21, 2026 How to Monitor Employer Reviews and Ratings Across Sites
Employer reviews shape who applies. How to monitor ratings and themes across review sites, benchmark competitors, and stay within each site's rules.
James Meadow · September 19, 2026 How to Build Large-Scale Training Datasets from the Open Web
Collecting pages is the easy part. How to turn an open-web crawl into a training corpus: frontier, opt-outs, extraction, filtering, dedup and decontamination.
Matt Brown · September 18, 2026 Building a B2B Lead Database with Web Scraping (Compliant & at Scale)
A lead database is an account model with contacts attached, not a list of emails. How to build one from public sources with compliance built into the schema.
Chris Collins · September 17, 2026 How to Handle Infinite Scroll and Dynamic Pagination When Scraping
Can't replay the hidden endpoint? Handle infinite scroll inside the render with scroll chains, load-more clicks and checks for virtualised lists.
Chris Collins · September 15, 2026 Moving Web Scraping API Data into SQL Databases at Scale
Fetching is the easy half. How to land scraping API output in SQL with idempotent loads, typed fields, schema-drift alerts and provenance intact.
Chris Collins · September 14, 2026 Aggregating Property Listings Across Global Real Estate Portals
Aggregating listings across portals and countries is a normalisation problem. Taxonomies, area units, price semantics and how to reconcile disagreement.
Chris Collins · September 13, 2026 Scraping Valuations, Rents & Mortgage Data for PropTech Platforms
Valuations, rents and mortgage rates are three different data problems with three different sources. How to collect each one and label what you collected.
Chris Collins · September 12, 2026