What to Check Before Troubleshooting Request Failure Rates?

The first step in troubleshooting request failure rates isn't logging — it's categorizing failures by HTTP status codes and exception types. Within the failure rate of a single scraping task, several types of failures usually coexist, and each type has different root causes and remediation paths. Investigating them all together is like fighting fires in every direction at once.

Before starting the investigation, build an observation matrix of the five symptom categories:

Count failed requests from the past 24 hours in your task and bucket them by these five symptoms. Whichever category has the highest share is what you fix first. Doing this step properly cuts subsequent investigation effort by at least half.

Is a Surge in 403s an IP Quality Issue or an IP Strategy Issue?

403 is both the most common symptom and the most easily misdiagnosed one. When teams see a surge of 403s, their first instinct is usually "this batch of IPs got blocked," and they rush to switch providers. But 403 has at least three root causes, and the provider-side one is only one of them.

First, IP quality layer. The IPs currently in use are from a shared pool, previously used for high-frequency scraping by other business teams, and have accumulated historical tags. The target site sees this IP history and returns 403 immediately. The fix path is switching to a more exclusive pool — dedicated proxies or tunnel pools with a cleaner anti-scraping history.

Second, request header layer. Combinations of User-Agent, Referer, Accept-Language and other headers are identified as typical crawler signatures by the target site. The fix path is rotating the UA pool, filling out header fields consistent with browser behavior, and adding a reasonable Referer chain. This layer of fix has nothing to do with IP strategy, but many teams misdiagnose it as an IP problem.

Third, IP strategy layer. The current IP session duration is too long — the same IP is observed making requests beyond the target site's threshold and enters a cumulative risk-control blacklist. The fix path is shortening the session duration, e.g., from 24 hours down to 5–10 minutes, so the IP rotates out before triggering cumulative risk control.

To distinguish among the three: grab a currently-403'd IP and manually open the target site in a browser. If it loads, it's an IP quality or strategy issue. If it doesn't load but other IPs in the same subnet do, that IP has been individually flagged. If a wide range of the same subnet can't access it, the entire IP block has been firewalled by the target site.

How to Locate the Rate-Limiting Layer When 429 Triggers Frequently?

429 tells you more precisely than 403 that you've been rate-limited, but it doesn't tell you which layer the limit is at. Target sites typically rate-limit at three layers: IP, account, and fingerprint. Misidentifying the layer leads you down a completely wrong fix path.

IP-layer rate limiting signal: switching IPs restores normal requests immediately. Fix path: lower the per-IP QPS (e.g., from 5 down to 2), or switch to a tunnel mechanism that rotates IPs per request.

Account-layer rate limiting signal: 429s persist after switching IPs, but a different account + same IP recovers. Fix path: expand the account pool, shorten the account rotation cycle, or lower the scraping frequency per account.

Fingerprint-layer rate limiting signal: switching IPs and accounts both fail to recover — you need to clear the browser fingerprint or TLS fingerprint. The fix path involves TLS/HTTP2 fingerprint spoofing, which falls under anti-detection engineering rather than pure IP strategy, and requires a fingerprint-spoofing toolchain.

429 also has a hidden signal: the Retry-After response header. If the target site returns this header, back off strictly according to its value — no need to switch IPs immediately. Ignoring Retry-After and switching IPs to retry will actually worsen the rate limiting.

When Request Timeouts Cluster, Is It Proxy-Side or Target-Side?

Timeouts are the most time-consuming symptom to investigate, because unlike 403/429 they don't clearly point to a layer. A timeout can come from the proxy side, the target site, or the link between them.

Proxy-side timeout signal: switching proxy providers noticeably drops the timeout rate. Fix path: evaluate the provider's link stability, especially peak-hour fluctuations. If you can't observe this yourself, compare timeout rates against the same target site across different proxy providers in your monitoring — obvious gaps mean it's a proxy-side issue.

Target-side timeout signal: direct access to the target site from your local network also times out. It might be a slow target server, CDN failure, or delayed responses to certain IP ranges. The fix isn't at the proxy strategy layer — you need to adjust request cadence, e.g., lower concurrency or raise the timeout threshold.

Link-layer signal: IPs from a specific region or ISP show abnormally high timeout rates while other regions are normal. Fix path: designate cities or ISPs at the proxy side. For example, public-opinion monitoring scenarios often use China Telecom nodes + specific city combinations to avoid link issues.

The timeout threshold itself is also a variable. The defaults of 5s connect / 10s response fit most scenarios, but for slow-responding target sites you may need to loosen it to 15–20s; during periods with high timeout rates, temporarily lowering the threshold can prevent request pileup.

How Do Empty Responses and Malformed JSON Tie Back to IP Strategy?

HTTP 200 with an empty body or malformed JSON structure is the most insidious kind of failure — the business code sees 200 and thinks it succeeded, while downstream consumes fake data. There are usually two root causes.

First, soft limits from the target site. The target site detects scraping signatures from this IP but doesn't want to signal that clearly with a 403, so it returns 200 with empty or fake data. Fix path: add response content validity checks — JSON must contain a specific key field, HTML must contain a specific class name — and treat validation failures as 403s, switching IPs immediately.

Second, cache penetration. Once request frequency from an IP exceeds the target site's cache refresh threshold, what comes back is the default value for expired cache, typically empty JSON or empty HTML. Fix path: lower per-IP request frequency, increase request intervals, or switch to a different IP block that isn't cache-penetrated.

Investigating empty responses is especially important because they don't show up in traditional failure-rate statistics. The business side needs to actively add a response validity check layer to expose these hidden failures.

How to Locate DNS Resolution Errors and Link Issues?

DNS resolution errors typically account for 1–3% of crawler failures — not much, but hard to investigate. Signals include inability to establish a connection, connections dropping immediately after being established, or resolving to the wrong IP.

Three typical scenarios:

• Proxy-side DNS config issues — the proxy provider's DNS servers return wrong IPs, especially DNS pollution in cross-border scraping. Fix path: explicitly specify DNS servers in your business code, or choose a proxy product that supports overseas DNS resolution.
• CDN switching — the target site switched CDN providers, and before DNS TTL expires, your business side is still hitting the old IP. Fix path: shorten DNS cache TTL, or periodically force DNS cache refresh.
• Active target-site protection — for suspicious IPs, the target site returns wrong DNS query results or delays responses. Fix path: switch to a higher-quality IP pool or avoid suspicious IP ranges.

Diagnostic tools for DNS issues: query directly with local dig or nslookup, then compare against proxy-side logs. Matching results but failing requests → link-layer issue. Non-matching results → DNS configuration issue.

FAQ

Q: If all five symptoms appear at once, which do I fix first? Fix the highest-share one first. If shares are close, order by remediation cost from low to high: request headers (part of 403), QPS adjustment (429), timeout thresholds, IP strategy adjustment (session duration, rotation mechanism), pool switching (highest cost). Fix low-cost, high-leverage problems first to quickly pull down overall failure rate, then look at the residual symptom distribution.

Q: Still getting 403 after switching IPs — is it the IP pool or something else? Most likely not the IP pool. Persistent 403s after IP swaps usually mean the request header layer was identified, UA is off, fingerprint leaked, or the account layer is limited. Run a controlled experiment: same IP + fresh UA + fresh account. If it works, it's an account or header issue; if it's still 403, it's IP quality or IP strategy.

Q: What should I do when the Retry-After response header appears? Back off strictly per its value; don't switch IPs. When a target site actively returns Retry-After, it's telling you when to come back — a relatively friendly form of rate limiting. Ignoring this header and retrying or IP-switching gets you classified as a protocol-violating crawler and triggers stricter limits or bans. Your code needs a dedicated branch for Retry-After.

Q: How do I distinguish empty responses / malformed JSON from legitimate "no data" business responses? Structural validation of response content, not length validation. Maintain an expected-field list in business code — list endpoints must have an items field, detail endpoints must have id and title. Failed validation counts as failure. Also keep raw responses in logs so you can trace back whether the target site truly returned no data.

Q: Timeout rates spiked — how do I quickly locate proxy-side vs. target-side? Run a 5-minute controlled test: send 100 requests direct from local to the target, and 100 through the proxy to the target. Compare timeout rates. Both high → target-side. Only proxy-side high → proxy-side. Only local direct high → possibly a local network issue. This test is very cheap and can determine direction within minutes of starting the investigation.