Scrapy Proxy Configuration: Middleware Setup Guide

Scrapy Proxy Setup

Configure proxies in Scrapy for efficient web scraping.

TL;DR: Create custom middleware or use scrapy-rotating-proxies. Set proxy in request.meta for per-request control.

Basic Configuration

settings.py

# Single proxy
HTTP_PROXY = 'http://user:pass@proxy.nobleproxy.com:8080'

DOWNLOADER_MIDDLEWARES = {
    'myproject.middlewares.ProxyMiddleware': 350,
}

middlewares.py

class ProxyMiddleware:
    def process_request(self, request, spider):
        request.meta['proxy'] = 'http://user:pass@proxy:port'

Rotating Proxies

Using scrapy-rotating-proxies

pip install scrapy-rotating-proxies

# settings.py
ROTATING_PROXY_LIST = [
    'http://proxy1:port',
    'http://proxy2:port',
    'http://proxy3:port',
]

DOWNLOADER_MIDDLEWARES = {
    'rotating_proxies.middlewares.RotatingProxyMiddleware': 610,
    'rotating_proxies.middlewares.BanDetectionMiddleware': 620,
}

Per-Spider Proxy

class MySpider(scrapy.Spider):
    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(
                url,
                meta={'proxy': 'http://proxy:port'}
            )

Best Practices

1. Implement retry logic
2. Detect bans and rotate
3. Use appropriate delays
4. Log proxy performance