Proxies for Scrapy

Check three things in order. First, the value must be a full URL — http://IP:port, not bare IP:port; Scrapy's HttpProxyMiddleware silently does nothing with a string it can't parse as a scheme. Second, make sure that middleware is actually running: if a custom entry in DOWNLOADER_MIDDLEWARES set scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware to None (a common copy-paste from rotation tutorials), nothing reads the meta key anymore. Third, if you're using scrapy-rotating-proxies, remember it deliberately skips requests that already have proxy in meta — that's a feature for exempting individual requests, but it surprises people who expect the middleware to override their manual value.

Usually not — BanDetectionMiddleware counts any non-200 outcome as a ban by default, so a target that answers probing with 403s or CAPTCHAs will get every address in your list quarantined within minutes, even though the proxies themselves are fine. Verify independently: curl -x http://IP:port https://example.com against a neutral site. If that returns cleanly, the target is blocking your pattern, not your addresses — slow the spider down (CONCURRENT_REQUESTS_PER_DOMAIN, AUTOTHROTTLE_ENABLED) or write a custom ban policy so soft responses from that site don't poison the whole pool. Genuinely dead addresses in a package are the rare case; when one stops answering from two different networks, we replace it — open a ticket.

Not natively — Scrapy's downloader is built on Twisted's HTTP client, and HttpProxyMiddleware speaks HTTP proxying only. Community workarounds exist (a local converter like Privoxy in front of the SOCKS address, or third-party download handlers), but they add a moving part to debug for no gain in a typical crawl. The practical answer: use the HTTP port. Every address we issue answers on HTTP, HTTPS, SOCKS4, and SOCKS5 simultaneously, so the same list serves your Scrapy project over HTTP and a SOCKS-native tool without buying anything twice.

Size it from the pool, not from your hardware. Scrapy defaults to 16 concurrent requests and 8 per domain — with a 100-address list that's harmless, but people raising CONCURRENT_REQUESTS to 500 against the same 100 addresses concentrate five simultaneous connections on each IP, and per-IP rate limits fire long before the crawl saturates bandwidth. A working starting point: keep total concurrency at or below your address count, cap CONCURRENT_REQUESTS_PER_DOMAIN at a single-digit number, and let AUTOTHROTTLE_ENABLED = True adapt from response times. On a rotating plan the calculus flips — the pool behind the gateway absorbs concurrency for you, and your ceiling is the plan's concurrent-connection allowance rather than a per-IP budget.

Either one works as your Scrapy proxy pool — the difference is who runs the rotation: with a package your middleware distributes requests across the exported list, with a rotating plan the switching happens behind our gateway. So count your monthly requests and do both sums — the crossover is lower than most people guess. Rotating plans start at $49/mo for 2.5 million requests — a spider fetching 80,000 pages a day lands almost exactly at that allowance. A $19/mo package with unlimited bandwidth handles the same volume for less — if your targets tolerate a fixed set of addresses between list refreshes. Rotating earns its premium when targets burn static IPs faster than three refreshes per 30 days can replace them, or when maximum IP diversity is the point. Steady, tolerant targets — package; hostile or diversity-hungry ones — rotating.