Screaming Frog Proxies

On Windows and Linux: File → Settings → Proxy. On macOS the same panel sits under Screaming Frog SEO Spider → Settings → Proxy. Tick “Use Proxy Server” and enter the address and port from your Dashboard list. If you authenticate by username and password, also tick “Use Proxy Credentials” and fill in both fields; with IP whitelisting you skip that checkbox entirely — add your machine's public IP in our Dashboard once (up to 5 source IPs, free) and the bare address works. Two details that catch people: the SEO Spider must restart before the proxy takes effect, so a “not working” proxy right after clicking OK is usually just a skipped restart; and the setting is application-wide — every crawl uses it until you turn it off, which is worth remembering before you crawl your own intranet through an external IP.

Work from cheap to expensive. First, the user-agent: WAFs recognize the “Screaming Frog SEO Spider” string, and Cloudflare's managed rules block it regardless of what IP it arrives from — switch to a Chrome preset under Configuration → User-Agent. Don't spoof Googlebot: Cloudflare verifies real Googlebot against Google's published IP ranges, so a fake one from your proxy is an instant block. Second, pace: lower the thread count under Configuration → Speed; Wordfence-style rate limits trip on velocity, not identity. Third, rendering: if the page loads in a browser but 403s in the crawl, try JavaScript rendering mode. And if it's your own site or a client's, stop fighting your own firewall — a skip-rule for your crawl IP in the WAF settings solves it permanently and works with any user-agent.

No — the proxy panel holds exactly one server, and there's no built-in rotation. For an SEO audit that's rarely a real limitation: a crawl is supposed to look like one consistent visitor, and a stable dedicated IP with a sensible speed setting survives most targets. If a specific site genuinely rate-limits any single address into uselessness, there's a trick that fits the one-server constraint: enter the gateway hostname of a rotating plan as that single proxy. Screaming Frog sees one server; the exit IP changes on every request on our side. Use it as the exception, not the default — some sites respond to seeing one “visitor” arrive from hundreds of addresses by escalating to CAPTCHAs, at which point the stable-IP-plus-patience approach was the better audit.

Usually the opposite — for sites you control, allowlist your office IP in the WAF and crawl directly at full speed; a proxy just adds a hop. The proxy case for your own properties is verification, not access: checking geo-redirects and hreflang behavior from the markets you serve, confirming that a country-specific promo actually shows in that country, or auditing how the site behaves for visitors your CDN treats as foreign. Teams running scheduled crawls from a VPS also route them through a dedicated address so the monitoring IP stays constant across office moves and ISP changes — the WAF allowlist entry never needs updating. For competitor and prospect audits, the calculus flips back to the standard one: their firewall doesn't know you, and the block should land on a disposable address, not your network.