Proxies for Web Scraping

Almost certainly the fingerprint layer. A proxy changes your IP reputation and spreads your request rate — but your HTTP client introduces itself before a single header is sent, through its TLS handshake. Python's requests and Node's axios produce handshake signatures (JA3/JA4) that no real browser emits, and Cloudflare or Akamai compare that signature against your claimed User-Agent at the edge: a Chrome User-Agent on a Python handshake is a contradiction, and the request dies before your proxy quality ever matters. The test: if a brand-new IP gets challenged on its very first request, the IP isn't the problem. Fixes, in ascending order of effort — a TLS-impersonation client (curl-cffi in Python, tls-client in Go), or a real browser via Playwright or Selenium, which carries a genuine fingerprint for free.

When sizing proxies for crawlers, work backwards from the pace one address can sustain. On most targets, staying under roughly 10 requests per minute per IP keeps you below rate-limit thresholds; strict sites want less, forgiving ones tolerate more. So: 1,000,000 pages a day is ~700 requests per minute overall, and at 7 per minute per address that's a 100-IP package — our smallest, at $19/mo. Double the volume or halve the safe pace, and you scale the package, not the architecture. Since billing is per IP with unlimited bandwidth, heavy pages cost the same as light ones — the only number that matters is request rate.

That's a soft block — the most expensive kind, because your scraper keeps running and keeps collecting nothing. Instead of an honest 403, the target serves a stripped page, a cached shell, or a JavaScript challenge disguised as content, so error-based retry logic never fires. Defend against it with content checks rather than status codes: assert that a known selector exists, that the response length is in the expected range, or that a price field actually parses as a number. When the check fails, treat it exactly like a block — retire that IP for the target, slow down, and inspect what changed. Sites that soft-block usually escalated from hard blocks deliberately; they're telling you your pattern is visible, not just your address.

Test before you spend. Run a probe of a few hundred requests through datacenter addresses: clean 200s with real content mean datacenter handles that target, and anything pricier is wasted budget — roughly half the public web falls in this category. Sustained 403s, instant CAPTCHAs, or soft blocks on first contact mean the target discounts datacenter ASNs on arrival, and that's what ISP proxies are for: the address belongs to a consumer ISP's network, so the origin check passes. The honest caveat — datacenter origin always remains visible to IP databases; no volume of rotation hides it. The common budget mistake runs the other way, though: paying residential rates for targets that never checked.

Refresh the list. Every package includes up to three free list refreshes per 30 days — you swap the addresses a target has learned to throttle for fresh ones from the pool, in the Dashboard, without a ticket. Between refreshes, the cheaper fix is routing: a rate-limited IP is only rate-limited for that target, so rotate it to a different site in your job queue and it keeps earning its keep. If your targets burn addresses faster than the refresh cadence, that's the signal to move that slice of traffic to a rotating plan, where the exit IP changes on every request and list management stops being your job.