What Variables Determine IP Rotation Frequency?

IP rotation frequency is not a number you can pull out of thin air. Setting the frequency is essentially about finding the balance point among three constraints.

Constraint 1: The target site's rate limiting window. Every site has a tolerance threshold for request density per IP. Exceeding it triggers restrictions. This threshold is typically calculated by time window, with common granularities at the second, minute, and hour level.

The thresholds above are industry reference values. Actual numbers vary as sites update their systems, so real-world testing is required before deployment.

Constraint 2: Safe request density per IP. Within a rotation cycle, the total requests sent from a single IP should stay within 60%–80% of the target site's threshold, leaving a safety margin. The reason is that multiple scraping tasks may share the same IP pool—keeping a single task within the threshold doesn't guarantee the IP's total request volume is within the threshold.

Constraint 3: Available IP pool capacity. The higher the rotation frequency, the more IPs consumed per unit of time. If the pool capacity can't support the configured rotation frequency, IPs will be reassigned prematurely, effectively negating the rotation.

The interplay of these three constraints can be expressed as a formula:

Minimum IP Pool Size = Concurrent Tasks × (Cooldown Time / Rotation Interval)

Assuming 10 concurrent tasks, a 5-minute cooldown per IP after use, and a 1-minute rotation interval, the minimum pool size = 10 × (5 / 1) = 50 IPs online simultaneously. If the pool only has 30 IPs, you either reduce concurrency or extend the rotation interval to at least 2 minutes.

How Should Rotation Frequency Be Set for Different Business Scenarios?

Different scenarios have vastly different requirements for data timeliness, request density, and access depth. Rotation frequency must be configured per scenario.

A common misconception: many teams set all tasks to "per-request rotation," assuming faster switching equals better safety. The actual result is often the opposite. Excessively high rotation frequency means each IP is discarded after a single request, draining the pool rapidly. Once the pool is depleted, newly assigned IPs may be "hot IPs" just used by other tasks, actually reducing request environment isolation.

In public opinion monitoring scenarios, monitoring targets are typically a fixed list of sites with collection cycles measured in hours or days. In this case, rotating every 10–30 requests, combined with 2–5 second intervals between requests, maintains a long-term stable collection channel without wasting IP resources.

How Should Enterprise-Grade Proxy Architecture Be Designed?

A single-machine direct-connect proxy architecture works for small-scale testing, but enterprise deployment requires a scheduling layer between the client and proxy service. The core architecture includes four components.

Component 1: Proxy Gateway Layer

The proxy gateway is the unified egress for all scraping requests, responsible for receiving business-side requests and dispatching them to the appropriate proxy channels. The gateway's core responsibilities include: request queuing, rate limiting, IP selection, and failure retry.

# Core scheduling logic for the proxy gateway (pseudocode)
class ProxyGateway:
    def __init__(self, config):
        self.pools = {}  # Pools isolated by business scenario
        self.rate_limiters = {}  # Rate limiting by target domain

    def get_proxy(self, task_id, target_domain):
        # 1. Select the corresponding IP pool based on task ID
        pool = self.pools[task_id]

        # 2. Check rate limit for the target domain
        limiter = self.rate_limiters.get(target_domain)
        if limiter and not limiter.allow():
            raise RateLimitError("Request too frequent, wait for next window")

        # 3. Get available IP from pool (skip IPs in cooldown)
        proxy = pool.get_available()
        if not proxy:
            raise PoolExhaustedError("IP pool exhausted, need to scale up or reduce concurrency")

        return proxy

    def release_proxy(self, proxy, success):
        # Return IP after request, set cooldown
        if success:
            proxy.start_cooldown(duration=300)  # 5-minute cooldown
        else:
            proxy.mark_failed()  # Failed IP enters inspection queue

Component 2: IP Pool Management Layer

The IP pool management layer handles the IP lifecycle: acquiring new IPs, assigning them to tasks, cooldown recovery, and retiring failed IPs. The key to enterprise deployment is pool isolation by business scenario.

The core value of pool isolation is preventing cross-contamination between tasks. If Task A's high-frequency requests cause certain IPs to be flagged by the target site, and Task B shares the same pool, those flagged IPs will affect Task B's success rate.

# IP pool configuration example (YAML format)
pools:
  sentiment_monitoring:  # Public opinion monitoring pool
    provider: tunnel_proxy
    min_pool_size: 50
    max_pool_size: 200
    rotation: request_count
    rotation_threshold: 20
    cooldown_seconds: 300
    target_domains:
      - "news.example.com"
      - "social.example.com"

  price_scraping:  # Price scraping pool
    provider: short_lived_proxy
    min_pool_size: 100
    max_pool_size: 500
    rotation: time_based
    rotation_interval_seconds: 45
    cooldown_seconds: 600
    target_domains:
      - "shop.example.com"

Component 3: Rate Limiting Layer

Rate limiting must operate simultaneously on two dimensions: per-IP and per-domain.

Per-IP rate limiting controls the total number of requests from a single proxy IP within the rotation cycle, preventing the IP from being blocked by the target site. Per-domain rate limiting controls the aggregate request rate from all IPs to the same domain, preventing the target site from detecting concentrated access patterns from the same proxy provider.

# Token bucket rate limiter
import time
import threading

class TokenBucketLimiter:
    def __init__(self, rate, capacity):
        self.rate = rate  # Tokens replenished per second
        self.capacity = capacity  # Bucket capacity
        self.tokens = capacity
        self.last_refill = time.time()
        self.lock = threading.Lock()

    def allow(self):
        with self.lock:
            now = time.time()
            elapsed = now - self.last_refill
            self.tokens = min(self.capacity, self.tokens + elapsed * self.rate)
            self.last_refill = now

            if self.tokens >= 1:
                self.tokens -= 1
                return True
            return False

# Usage example: max 20 requests per minute
limiter = TokenBucketLimiter(rate=20/60, capacity=5)

The advantage of the token bucket algorithm is that it allows short bursts of requests while maintaining the long-term average rate within the threshold. The capacity parameter determines the maximum burst size allowed.

Component 4: Monitoring and Alerting Layer

Enterprise deployment must include real-time monitoring. Core metrics include:

How to Choose Among the Three Rotation Strategy Modes?

IP rotation has three fundamental modes, each suited to different scenarios.

Mode 1: Time-Based Rotation

Switch IPs at fixed time intervals regardless of how many requests were sent during that period. Best for scenarios with relatively stable request frequency.

# Time-based rotation scheduling logic
import time

class TimeBasedRotator:
    def __init__(self, interval_seconds, pool):
        self.interval = interval_seconds
        self.pool = pool
        self.current_proxy = None
        self.last_switch = 0

    def get_proxy(self):
        now = time.time()
        if self.current_proxy is None or (now - self.last_switch) >= self.interval:
            if self.current_proxy:
                self.pool.release(self.current_proxy)
            self.current_proxy = self.pool.acquire()
            self.last_switch = now
        return self.current_proxy

The advantage of time-based rotation is simplicity and predictable IP consumption. The downside is uneven IP utilization during traffic fluctuations—IPs are wasted during low periods, and per-IP request density may exceed limits during peaks.

Mode 2: Request-Count-Based Rotation

Switch IPs after every N requests. Best for scenarios with fluctuating request frequency.

class RequestCountRotator:
    def __init__(self, max_requests, pool):
        self.max_requests = max_requests
        self.pool = pool
        self.current_proxy = None
        self.request_count = 0

    def get_proxy(self):
        if self.current_proxy is None or self.request_count >= self.max_requests:
            if self.current_proxy:
                self.pool.release(self.current_proxy)
            self.current_proxy = self.pool.acquire()
            self.request_count = 0
        self.request_count += 1
        return self.current_proxy

The advantage of request-count-based rotation is that regardless of request rate changes, each IP's request load stays evenly distributed. In app analytics scenarios, API request frequency fluctuates with user behavior, making request-count-based rotation more stable than time-based rotation.

Mode 3: Session-Based Rotation

Each scraping session is bound to a fixed IP, which is released when the session ends. Best for scenarios requiring login state or Cookie consistency.

In production environments, the three modes can be combined. For example, the primary strategy uses request-count-based rotation with a time-based ceiling as a safeguard: regardless of whether the request count threshold is reached, an IP must be rotated after being held for more than 10 minutes.

How to Tune Frequency After Deployment?

Frequency tuning is a continuous iterative process, broken into three phases.

Phase 1: Baseline Measurement (1–3 Days Before Launch)

Use low frequency and low concurrency for probing. Record the target site's actual tolerance threshold.

# Baseline measurement script: gradually increase frequency, record success rate changes
for interval in 10 8 5 3 2 1; do
  success=0
  total=100
  for i in $(seq 1 $total); do
    code=$(curl -s -o /dev/null -w '%{http_code}' \
      -x "http://proxy:port" \
      "https://target-site.com/api/data")
    [ "$code" -eq 200 ] && success=$((success + 1))
    sleep "$interval"
  done
  rate=$((success * 100 / total))
  echo "Interval ${interval}s: Success rate ${rate}%"
done

When the success rate drops noticeably from above 95%, the corresponding request interval marks the threshold boundary. Production frequency should be set at 60%–80% of the threshold.

Phase 2: Gradual Ramp-Up (Week 1 After Launch)

Start at 20% of target concurrency, increase by 20% daily, while monitoring success rate and IP pool consumption.

Phase 3: Steady-State Operations

Once stable after launch, establish a regular inspection routine. Target site anti-bot policies are updated periodically, and rotation parameters need to be adjusted accordingly.

It's recommended to re-run a baseline test every two weeks using the Phase 1 probing script to re-measure thresholds and compare with previous results. If the threshold drops by more than 20%, the target site has tightened its policies, and you need to lower request frequency or increase IP pool capacity accordingly.

In large-scale web scraping deployments, anti-bot policies vary significantly across target sites, and a uniform frequency configuration cannot cover all scenarios. The recommended approach is to configure by target domain group, with each group having its own rotation frequency and IP pool parameters, sharing the underlying proxy infrastructure from providers like qg.net while isolating scheduling strategies.

FAQ

Q: Are IP rotation frequency and request interval the same thing?

No. Request interval is the time gap between two HTTP requests, controlling the request rate. IP rotation frequency is how often you switch proxy IPs, controlling the request density per IP. A single IP can send multiple requests within a rotation cycle, with a fixed interval between each. The two must be configured together: request interval determines the number of requests per IP per unit of time, and rotation frequency determines when to switch to a new IP after that count accumulates.

Q: Why does my success rate drop when I increase rotation frequency?

The most common cause is IP reuse after the pool is rapidly depleted. High-frequency rotation consumes IPs faster than the pool can replenish, causing IPs still in cooldown to be reassigned to new tasks. Another cause is that some target sites detect a large number of different IPs from the same network segment within a short period—high-frequency rotation actually exposes the scale signature of scraping activity.

Q: What's the difference between tunnel proxies and API-extracted proxies in terms of rotation mechanism?

With tunnel proxies, IP rotation is handled automatically by the proxy server. Each request through the tunnel is automatically assigned a different exit IP, with rotation frequency determined by server-side configuration. With API-extracted proxies, rotation is initiated by the client, which calls an API to obtain a new IP and manages IP usage and switching itself. Tunnel proxies are suited for scenarios that don't require fine-grained control over IP allocation. API-extracted proxies are suited for scenarios that require custom rotation strategies based on business logic.

Q: How do you prevent interference when multiple scraping tasks share an IP pool?

The most effective approach is pool isolation by task, with each task using its own independent IP sub-pool. If IP resources are limited and full isolation isn't feasible, the fallback is IP tagging within the shared pool: tag each IP with the task ID and target domain of its most recent use, and prioritize assigning IPs that haven't been used by tasks targeting the same domain. Pool isolation costs more in total IP usage but provides fault isolation between tasks.

Q: How long should the cooldown period be?

Cooldown duration depends on how long the target site retains access records. Most sites use sliding windows between 5–15 minutes, so setting cooldown to 1.5–2× the window length is a safe bet. For example, if the target site's detection window is 5 minutes, set cooldown to 8–10 minutes. If you can't determine the detection window length, 300 seconds is a relatively conservative default value with broad applicability.