“Your proxies are slow” is one of the most common messages we get from customers. Sometimes it is true. More often, when we look at the scraper together, most of the time goes on things the scraper does to itself: opening a new connection for every page, parsing with a slow library, waiting on one request at a time.
To show how much that matters, we took a simple scraper, ran it through our residential proxies, and changed one thing at a time. The same 120 pages went from 298 seconds to 40, a 7.5 times speed-up, without changing anything about the proxy. This guide walks through that test, the tool we used to find the slow parts, and a checklist for diagnosing your own scraper.
Key takeaways
- Measure before you change anything. A stage timer that records how long fetching, parsing and writing take will tell you where the time goes in minutes.
- Reuse connections. Opening a new connection through a proxy for every request more than tripled the time per fetch in our test. One
Sessionfixed it. - Parsing can cost as much as fetching. Switching the HTML parser cut parsing time by more than half.
- Concurrency gives the biggest win, then exposes the next bottleneck. Eight workers halved the run time, and showed that CPU-bound parsing had become the limit.
- Not everything slow is worth fixing. Database writes took 0.3 seconds out of 298, so batching them changed nothing.
The test
We collected 120 English Wikipedia articles through Shifter’s residential proxies with US exits, from one machine in Europe. Wikipedia articles are large pages, often several hundred kilobytes of HTML. For every page the scraper fetches the HTML, parses it with Beautiful Soup to read the title and count the links, and writes a row to SQLite.
We ran the same job five times, changing one thing each time. The scraper identified itself with a descriptive User-Agent, which Wikipedia requires (its servers reject the default Python User-Agent), and retried a failed request up to twice.
| Version | Change | Total time | Fetch per page | Parse per page |
|---|---|---|---|---|
| 1. Baseline | New connection per request, html.parser, one at a time | 298 s | 1,906 ms | 577 ms |
| 2. Reuse connections | One requests.Session | 125 s | 568 ms | 473 ms |
| 3. Faster parser | lxml instead of html.parser | 76 s | 426 ms | 205 ms |
| 4. Concurrency | 8 worker threads | 40 s | 551 ms | 2,024 ms |
| 5. Batch writes | One transaction instead of one per row | 42 s | 579 ms | 2,141 ms |
Fetch times through residential exits vary from run to run by a hundred milliseconds or so; the differences above are far larger than that.
Step one: measure where the time goes
The tool is a small timer that adds up wall-clock time per stage, safely across threads:
import time
from collections import defaultdict
from contextlib import contextmanager
from threading import Lock
class StageTimer:
"""Adds up wall-clock time per pipeline stage, safely across threads."""
def __init__(self):
self.totals = defaultdict(float)
self.counts = defaultdict(int)
self._lock = Lock()
@contextmanager
def stage(self, name):
start = time.perf_counter()
try:
yield
finally:
elapsed = time.perf_counter() - start
with self._lock:
self.totals[name] += elapsed
self.counts[name] += 1
def report(self, wall_seconds):
busy = sum(self.totals.values())
print(f"wall {wall_seconds:.1f}s, stage time {busy:.1f}s")
for name, total in sorted(self.totals.items(), key=lambda kv: -kv[1]):
print(f"{name:>8}: {total:7.1f}s {100 * total / busy:5.1f}% "
f"{1000 * total / self.counts[name]:7.1f} ms per call")Wrap each stage of your scraper in with timer.stage("fetch"):, with timer.stage("parse"): and so on, and print the report at the end. It costs almost nothing to run and turns “the scraper is slow” into a specific answer.
Here is the full test scraper with the timer in place. The run() function takes the four settings we changed:
import os
import sqlite3
import time
from concurrent.futures import ThreadPoolExecutor
import requests
from bs4 import BeautifulSoup
from stagetimer import StageTimer
from urls import URLS
user, password = os.environ["PROXY_USER"], os.environ["PROXY_PASS"]
PROXY = f"http://{user}-country-us:{password}@p.shifter.io:443"
PROXIES = {"http": PROXY, "https": PROXY}
HEADERS = {"User-Agent": "ExampleScraper/1.0 (+https://example.com/bot)"} # name your scraper
def scrape(url, session, parser, timer):
with timer.stage("fetch"):
for attempt in range(3):
try:
if session is None: # a new connection for every request
response = requests.get(url, headers=HEADERS, proxies=PROXIES, timeout=20)
else: # reuse connections through one Session
response = session.get(url, timeout=20)
response.raise_for_status()
break
except requests.RequestException:
if attempt == 2:
raise
html = response.text
with timer.stage("parse"):
soup = BeautifulSoup(html, parser)
title = soup.find("h1").get_text(strip=True)
links = len(soup.select("a[href]"))
return url, title, links
def run(reuse=False, parser="html.parser", workers=1, batch_writes=False):
timer = StageTimer()
session = None
if reuse:
session = requests.Session()
session.proxies = PROXIES
session.headers.update(HEADERS)
adapter = requests.adapters.HTTPAdapter(pool_connections=workers, pool_maxsize=workers)
session.mount("https://", adapter)
db = sqlite3.connect("pages.db", check_same_thread=False)
db.execute("CREATE TABLE IF NOT EXISTS pages (url TEXT, title TEXT, links INTEGER)")
db.execute("DELETE FROM pages")
db.commit()
start = time.perf_counter()
with ThreadPoolExecutor(workers) as pool:
rows = pool.map(lambda u: scrape(u, session, parser, timer), URLS)
if batch_writes:
rows = list(rows)
with timer.stage("write"):
db.executemany("INSERT INTO pages VALUES (?, ?, ?)", rows)
db.commit()
else:
for row in rows:
with timer.stage("write"):
db.execute("INSERT INTO pages VALUES (?, ?, ?)", row)
db.commit() # one transaction per row
wall = time.perf_counter() - start
timer.report(wall)
return wall
if __name__ == "__main__":
steps = [
("baseline", {}),
("reuse connections", {"reuse": True}),
("faster parser", {"reuse": True, "parser": "lxml"}),
("8 workers", {"reuse": True, "parser": "lxml", "workers": 8}),
("batch writes", {"reuse": True, "parser": "lxml", "workers": 8, "batch_writes": True}),
]
for name, config in steps:
print(f"\n== {name}")
run(**config)urls.py holds the list of 120 article URLs. Running it printed:
== baseline
wall 298.2s, stage time 298.1s
fetch: 228.7s 76.7% 1905.7 ms per call
parse: 69.2s 23.2% 576.6 ms per call
write: 0.3s 0.1% 2.2 ms per call
== reuse connections
wall 125.0s, stage time 125.2s
fetch: 68.2s 54.5% 568.3 ms per call
parse: 56.7s 45.3% 472.7 ms per call
write: 0.3s 0.2% 2.1 ms per call
== faster parser
wall 75.7s, stage time 75.9s
fetch: 51.1s 67.4% 425.8 ms per call
parse: 24.6s 32.4% 204.8 ms per call
write: 0.2s 0.3% 1.6 ms per call
== 8 workers
wall 39.7s, stage time 309.8s
parse: 242.8s 78.4% 2023.6 ms per call
fetch: 66.1s 21.3% 551.0 ms per call
write: 0.8s 0.3% 6.9 ms per call
== batch writes
wall 42.1s, stage time 326.4s
parse: 256.9s 78.7% 2141.0 ms per call
fetch: 69.5s 21.3% 579.0 ms per call
write: 0.0s 0.0% 1.8 ms per callWhat each change did
Reusing connections cut fetch time by 70%. In the baseline, every call to requests.get() opens a brand-new connection. Through a proxy, that means a TCP connection to the proxy, a tunnel request, and a full TLS handshake with the target through the exit, before a single byte of the page arrives. A requests.Session keeps connections open and reuses them, so most requests skip all of that setup. Fetch time per page fell from 1.9 seconds to 0.57. Almost every HTTP library has an equivalent: a client object that you create once and share.
A faster parser halved parsing time. Beautiful Soup can use several parsers. The built-in html.parser is pure Python; lxml is written in C. Switching cut parse time from 473 to 205 milliseconds per page on these large pages, with one changed argument. If you only need a few fields, selecting them with lxml directly or extracting embedded JSON is faster still.
Concurrency halved the total time and moved the bottleneck. Most of a fetch is waiting for the network, and threads can wait in parallel. With eight workers, the run fell from 76 to 40 seconds. But look at the parse column: 205 milliseconds per page became over 2 seconds. In CPython, threads share one interpreter lock, so CPU-heavy work such as parsing does not run in parallel; eight threads queue up for the same core. The stage timer makes this visible: parsing went from a third of the stage time to almost 80%. At that point the scraper is limited by CPU, and the next step is to parse in separate processes or to parse less.
Batching writes changed nothing, because writes were never the problem: 0.3 seconds out of 298 in the baseline. The small difference between versions 4 and 5 is run-to-run noise. This is the point of measuring first. Batching database writes is good advice in general, and it would have been wasted effort here.
A checklist for a slow scraper
| What you see | Likely cause | What to check |
|---|---|---|
| Every request is slow, CPU is idle | One request at a time | Add concurrency; most scrapers spend their time waiting |
| The first request to each host is much slower than the rest | Connection setup | Reuse a session or client; check that keep-alive is not disabled |
| CPU is at 100% on one core | Parsing or other CPU-bound work | Use a faster parser, extract less, or move parsing to processes |
| Throughput drops as concurrency rises | The target is throttling you, or your own machine is saturated | Watch status codes and response times per target; see our guide to planning request volume and concurrency |
| Occasional requests take 20 to 30 seconds | Timeouts and retries | Set explicit timeouts, cap retries, and log them; see retry and backoff for web scrapers |
| The scraper slows down over hours | Queues growing, memory pressure, unbounded buffers | Add backpressure; see backpressure and flow control |
| Pages are slow and heavy | Rendering a browser when the HTML would do | Fetch HTML directly, or block images, media and fonts |
When it really is the proxy
Sometimes the proxy is the bottleneck, and it is worth knowing how to tell.
- Compare against a direct baseline. Time the same requests without the proxy from the same machine. If both are slow, the problem is the target or your code.
- Look at fetch time after connection reuse. If fetches are still slow with a reused session and the target is fast directly, the route through the proxy is adding the time.
- Check your targeting. Narrow filters, such as a specific city, leave fewer exits to choose from. Exits far from the target add round-trip time to every request.
- Watch percentiles, not averages. A few very slow exits can drag an average up while most requests are fine.
Our guide to reducing latency with residential proxies covers the proxy-side settings in detail. If you have measured and the numbers point at the proxy, send them to our support team: stage timings like the ones above make a slow-proxy report much faster to investigate.
FAQ
Why is my scraper slow with proxies?
The most common causes are in the scraper itself: opening a new connection for every request, fetching one page at a time, and slow parsing. In our test, fixing those made the same job 7.5 times faster without changing the proxy.
Does connection reuse work through a proxy?
Yes. A requests.Session with proxies set keeps connections open, which saves the connection and TLS setup on each request. In our test it cut fetch time per page from 1.9 seconds to 0.57.
How many threads should a scraper use?
Enough to keep the network busy, within what the target and your plan allow. Measure as you increase it: when parse time per page or error rates start to climb, you have found the limit. CPU-bound work such as parsing does not speed up with more threads in CPython.
Is BeautifulSoup slow?
Beautiful Soup itself is a convenience layer; most of its speed depends on the parser underneath. With lxml instead of the built-in html.parser, parsing was more than twice as fast in our test.
The bottom line
Before blaming the proxy, measure the scraper. A stage timer shows within minutes whether the time goes on fetching, parsing or writing. In our test, reusing connections, switching parsers and adding concurrency made the same job 7.5 times faster with the same proxy. Fix the biggest stage first, measure again, and stop when the remaining time really is the network.