How to set a proxy in Scrapy

Scrapy ships with proxy support built in: HttpProxyMiddleware is enabled by default, so there's nothing to install. You can set a proxy through environment variables, per request via meta['proxy'], or — the pattern that fits scraping best — with a small middleware that rotates proxies from your list. Credentials go straight into the proxy URL, which also answers the authentication question.

Before you start: proxy details and ports

You'll need your proxy IP address and port. Both are listed in your Dashboard — open your active subscription to see the full connection details. Scrapy works with HTTP proxies only (there's no built-in SOCKS support), so you need one of the two HTTP ports:

Authentication	HTTP port	Proxy URL format
Username / password credentials from your Dashboard	`8080`	`http://login:password@IP:8080`
IP whitelisting no login required	`8085`	`http://IP:8085`

With IP whitelisting, you first add your server's IP address to the whitelist in the Dashboard. After that, the proxy accepts requests from that machine with no credentials — the URLs get shorter and no secrets end up in your project.

The quick way: environment variables

HttpProxyMiddleware respects the standard http_proxy and https_proxy variables — set both to the same proxy and every request in the crawl goes through it, with no code changes:

export http_proxy="http://your-login:[email protected]:8080" export https_proxy="http://your-login:[email protected]:8080" scrapy crawl myspider

Note that https_proxy covers requests to https:// URLs — which is most of the modern web — and its value still starts with http://: it's the same proxy, reached over the same protocol.

Per-request: meta['proxy']

For control from inside the spider, pass the proxy in the request's meta — it takes precedence over the environment variables:

import scrapy class IpCheckSpider(scrapy.Spider): name = "ipcheck" def start_requests(self): yield scrapy.Request( "https://api.ipify.org", meta={"proxy": "http://your-login:[email protected]:8080"}, ) def parse(self, response): self.logger.info("Exit IP: %s", response.text) # prints the proxy IP

This spider doubles as a connection test: run it and the log shows the proxy IP instead of your own.

Credentials belong in the meta['proxy'] URL, not in a hand-crafted Proxy-Authorization header. Since Scrapy 2.6.2 the middleware manages that header itself — including wiping it when a redirect or retry switches proxies, so your credentials can't leak to a third party. If the password contains special characters, percent-encode them (@ → %40).

Loading proxies from a list and rotating them

With a proxy package, nobody types addresses into code by hand: export your full list from the Dashboard to a file — one IP:port per line, the field order and delimiter are configurable at export — and let a downloader middleware assign a proxy to every request:

middlewares.py

import random # Port 8080: set the credentials once — they apply to every address in the list. # Port 8085 with IP whitelisting: set both to None. LOGIN, PASSWORD = "your-login", "your-password" class ProxyListMiddleware: def __init__(self): auth = f"{LOGIN}:{PASSWORD}@" if LOGIN else "" # proxies.txt — one address per line: 123.45.67.89:8080 with open("proxies.txt") as f: self.proxies = [f"http://{auth}{line.strip()}" for line in f if line.strip()] def process_request(self, request, spider): request.meta["proxy"] = random.choice(self.proxies)

settings.py

DOWNLOADER_MIDDLEWARES = { "myproject.middlewares.ProxyListMiddleware": 350, }

The order value just needs to be below 750 — that's where the built-in HttpProxyMiddleware sits, and it must run after your middleware to pick up the meta['proxy'] value. Every request now gets a random IP from your list; swap random.choice for itertools.cycle if you prefer strict round-robin. Since the credentials travel inside the proxy URL, one pair of constants covers the whole list — and if your exported file already embeds them (login:password@IP:port lines, the Dashboard export supports any field layout), drop the constants and use each line as is.

Our packages start at 100 IPs, so the list itself isn't the bottleneck — and when a target starts rate-limiting, refresh the list on request and re-export the file. Which of our proxy types fits Scrapy crawls best, with pricing and specs, is broken down on the proxies for Scrapy page. If you'd rather not manage rotation in code at all, a rotating proxy flips the logic: you keep one gateway address in meta['proxy'], and the exit IP rotates on every request.

What about SOCKS5?

Scrapy's downloader is built on Twisted and speaks to HTTP proxies only — there's no native SOCKS5 support, and the workarounds (local forwarders that convert HTTP to SOCKS) add a moving part you don't need. Every PapaProxy.net package includes HTTP, HTTPS, SOCKS4, and SOCKS5 access on the same IPs, so for Scrapy simply use the HTTP ports from the table above.

If something doesn't work

The most common causes: a 407 Proxy Authentication Required response means the credentials are wrong, missing from the URL, or the machine's IP isn't whitelisted on port 8085; TunnelError or ConnectionRefusedError usually means the port doesn't match the authentication method (see the table); requests to https:// sites going around the proxy means https_proxy wasn't set alongside http_proxy; and if your rotation middleware seems ignored, check that its order in DOWNLOADER_MIDDLEWARES is below 750. If none of these is it, message us in live chat right on this page — we'll help you get connected.