Scrapy-Redis handles distributed task queues and request deduplication, but proxy scheduling requires a separate, centralized proxy pool. A practical architecture manages the request queue, proxy pool, leases, failure statistics, and cooldown queue independently. A downloader middleware then assigns proxies dynamically and decides whether to reuse, deprioritize, or cool down each proxy based on the response.

When a distributed scraper expands from one machine to multiple servers, the first problems are often not caused by the Redis queue. More commonly, the same proxies are reused across workers, failed proxies keep being reassigned, and failed requests cannot be traced to the responsible worker. Scrapy-Redis allows multiple crawler processes to share a request queue, but it does not automatically acquire, allocate, or recycle proxies.

Proxy scheduling should therefore be designed as an independent resource layer.

Why Can't Scrapy-Redis Handle Proxy Scheduling by Itself?

Scrapy-Redis distributes tasks; it does not manage outbound network resources. It can share pending requests, deduplicate URLs, and resume interrupted crawls, but every crawler process must still decide which proxy to use for each request.

A complete cluster usually includes the following components:

If proxies are selected randomly inside process_request, multiple workers may receive the same failed proxy. If proxies are configured only in settings.py, the cluster cannot update their shared status consistently after scaling out.

How Should a Distributed Cluster Separate Tasks from Proxy State?

The request queue and the proxy pool should be separate state systems. Do not mix proxy strings directly into the URL queue.

The cluster can follow this structure:

                    +----------------------+
                    | Redis Request Queue  |
                    | URLs, params, retries|
                    +----------+-----------+
                               |
                  +------------+------------+
                  |                         |
         +--------v---------+      +--------v---------+
         | Crawler Worker A |      | Crawler Worker B |
         | Scrapy-Redis     |      | Scrapy-Redis     |
         +--------+---------+      +--------+---------+
                  |                         |
                  +------------+------------+
                               |
                    +----------v-----------+
                    | Central Proxy Layer  |
                    | Acquire, lease, return|
                    +----------+-----------+
                               |
                  +------------+------------+
                  |                         |
         +--------v---------+      +--------v---------+
         | Available Queue  |      | Cooldown Queue   |
         | proxy:ready      |      | proxy:cooldown   |
         +------------------+      +------------------+

Use separate Redis keys for each type of state:

Do not store a proxy as a bare string only. In a production system, each proxy record should include at least its address, protocol, region, expiration time, and latest status. For example:

{
  "host": "proxy.example.com",
  "port": 8000,
  "scheme": "http",
  "region": "cn-east",
  "expires_at": 1790000000,
  "fail_count": 0
}

How Should Proxies Be Acquired and Returned?

Proxy allocation needs both an acquisition action and a return action. A worker should not simply remove a proxy for every request and never return it.

The simplest implementation uses two queues:

import redis


class RedisProxyPool:
    def __init__(self, url="redis://127.0.0.1:6379/0"):
        self.redis = redis.Redis.from_url(url, decode_responses=True)
        self.ready_key = "proxy:ready"
        self.leased_key = "proxy:leased"

    def acquire(self):
        # Atomically move a proxy from the ready queue to the lease queue.
        proxy = self.redis.rpoplpush(
            self.ready_key,
            self.leased_key,
        )
        return proxy

    def release(self, proxy, healthy=True):
        if not proxy:
            return

        self.redis.lrem(self.leased_key, 1, proxy)

        if healthy:
            self.redis.rpush(self.ready_key, proxy)
        else:
            self.redis.zadd(
                "proxy:cooldown",
                {proxy: 60},
            )

RPOPLPUSH moves a record between queues during acquisition, reducing the chance that multiple workers will receive the same proxy. A production deployment must also handle two edge cases:

1. A proxy in the lease queue must not be lost permanently if a worker exits unexpectedly.
2. A recovery job must check proxies in the cooldown queue and return eligible proxies to the available queue.

Cooldown duration should not be fixed. It can increase with consecutive failures: 30 seconds after the first failure, 60 seconds after the second, and temporary suspension after the failure count reaches a defined threshold.

How Does a Scrapy Downloader Middleware Connect to the Proxy Pool?

The downloader middleware injects a proxy before a request is sent, then updates the proxy state after receiving a response or exception.

The following is a simplified example:

from scrapy import signals
from scrapy.exceptions import IgnoreRequest


class DistributedProxyMiddleware:
    def __init__(self, proxy_pool):
        self.proxy_pool = proxy_pool

    @classmethod
    def from_crawler(cls, crawler):
        pool = RedisProxyPool(
            crawler.settings.get("REDIS_URL")
        )
        return cls(pool)

    def process_request(self, request, spider):
        if request.meta.get("proxy"):
            return None

        proxy = self.proxy_pool.acquire()

        if not proxy:
            # In production, delay the retry or put the request back in the queue.
            request.meta["proxy_unavailable"] = True
            raise IgnoreRequest("no proxy available")

        request.meta["proxy"] = proxy
        request.meta["proxy_lease"] = proxy
        return None

    def process_response(self, request, response, spider):
        proxy = request.meta.get("proxy_lease")

        if response.status in (200, 206):
            self.proxy_pool.release(proxy, healthy=True)
        elif response.status in (407, 403, 429, 502, 503):
            self.proxy_pool.release(proxy, healthy=False)
        else:
            self.proxy_pool.release(proxy, healthy=True)

        return response

    def process_exception(self, request, exception, spider):
        proxy = request.meta.get("proxy_lease")
        self.proxy_pool.release(proxy, healthy=False)
        return None

This example demonstrates the scheduling relationship. A production implementation should also:

• Prevent synchronous Redis operations from blocking Scrapy's event loop.
• Put requests back into the task queue when no proxy is available instead of dropping them.
• Distinguish proxy failures from failures caused by the target website.
• Store each request's retry count in meta to prevent infinite retries.
• Periodically scan expired leases and recycle their proxies when a worker exits.

How Can You Distinguish a Failed Proxy from a Target-Site Failure?

Do not classify every non-200 response as a proxy failure. Evaluate the error type and repeated results across multiple samples.

A three-level assessment can be used:

The proxy pool can maintain a simple score for each proxy:

score = successful requests / total requests

The score must be evaluated within time windows. Track data for the last 10 minutes, the last hour, and the current day so that earlier successes do not hide the fact that a proxy has recently stopped working.

How Should Concurrency Match the Number of Proxies?

Concurrency should not be determined only by the number of crawler workers. It must also reflect the number of available proxies, the proxy reuse interval, and the target site's acceptable request rate.

Start with a rough capacity estimate:

target concurrency per proxy ~= total concurrency / concurrently available proxies

For example, if the cluster has a total concurrency of 60 but only 20 stable proxies, each proxy handles about three concurrent requests on average. This is only a capacity estimate; it does not mean the target website will permit that request rate.

During tuning, monitor:

• Concurrent requests assigned to each proxy.
• The interval between consecutive requests through the same proxy.
• Changes in success rate after proxy rotation.
• Connection timeout rate and response time.
• The percentage of 429 and 403 responses from the target.
• Redis request queue length and consumption rate.

When the failure rate increases, first reduce per-worker concurrency and extend retry intervals. Do not respond by immediately multiplying the number of failed or low-quality proxies. Adding more proxies can relieve resource shortages, but it cannot fix incorrect request parameters, broken parsing logic, or an excessive request rate.

Which Metrics Should Distributed Proxy Scheduling Monitor?

At minimum, monitor task state, proxy state, and business results together.

Do not show only a "proxy availability rate" on the monitoring dashboard. It is more useful to correlate proxy status with the valid-data rate because a successful proxy connection does not guarantee a valid scraping result.

Conclusion

Proxy scheduling for a distributed scraper should be built independently of the Scrapy-Redis request queue. A centralized proxy pool, lease management, failure cooldowns, and business-level monitoring provide unified control over both task distribution and outbound proxy resources.

FAQ

Q: Should every worker in a Scrapy-Redis cluster maintain its own proxy pool?

No. Fully independent proxy pools make it easy to assign the same proxy more than once, prevent workers from sharing failure state, and lead to uneven resource usage. A better design keeps proxy state centralized while workers only acquire and return proxies.

Q: Should requests be dropped when the proxy pool has no available IPs?

No. Delay and requeue the request, or place it in a separate waiting queue with a maximum wait time. After the request exceeds its retry limit, record it in a failure queue for later inspection.

Q: How many concurrent requests should one proxy handle?

There is no universal value. Load-test the system based on the proxy type, target response time, request size, and target access rules. Start with low concurrency, monitor connection timeouts, the 429 response rate, and the valid-data rate, then increase concurrency gradually.

Q: Should a proxy always be blocked after it receives a 403 response?

Not necessarily. A 403 response may be caused by request parameters, request rate, or session state. Move a proxy to cooldown or invalid status only when it repeatedly produces connection errors or authentication failures across multiple requests and time windows.

Q: Can Scrapy-Redis automatically recover proxies leased by a failed worker?

Not completely. If a worker exits after moving a proxy into the lease queue, you need a lease expiration time, heartbeat mechanism, or recovery job to return stale proxies to the available queue.