10 Best Fixes for HTTP 429 Errors in Web Scraping (2026)

Your job queue is dead in the water, and it's not because of a logic bug. It's because the server determined you've sent too many requests. That's the blunt reality behind the HTTP 429 Too Many Requests status code: a hard stop, not a suggestion, triggered the moment traffic crosses an invisible threshold.

You added a sleep(3), maybe even a randomized jitter, and still kept hitting HTTP 429 errors. That's because production rate limiters rarely count with simple fixed windows. Many run token bucket algorithms that allow short bursts and still cap the average rate, so a steady, persistent stream gets cut off just as surely as a spike. A flat delay can't adapt to that.

So here's the answer upfront: stop pacing requests like a metronome. Read what the server tells you, match the shape of its bucket, and spread load across whatever it limits by: IP, credential or endpoint. The ten fixes follow roughly that order. Fixes 1 to 3 change how a single client reads and paces against the server. Fixes 4 and 5 keep one bad endpoint from stalling the rest. Fixes 6 to 9 spread load and handle layered or transport-level limits. Fix 10 is what Anakin handles so you don't have to build it.

Distributed IP pools, credentialed sessions and browser-based execution become standard tooling once volume gets real, and Anakin.io covers all three under one API key: URL Scraper for reads, Wire for authenticated actions against a specific site, Browser API for full interactive sessions. A few techniques other scraping platforms recommend are worth borrowing too, and they're credited where they appear.

Key Takeaways

Six rules do most of the work in the fixes below.

Rate limits model burst capacity, not just throughput: A token bucket specification TB(r,Bmax) includes a token fill rate of r tokens/sec and a bucket capacity Bmax. Uniform delays ignore that reserve allowance.

Parse Retry-After exactly: A 429 response often carries a Retry-After header, defined in RFC 9110, in two forms: a number of seconds to wait, or an HTTP date to wait until. Blind exponential backoff is a guess. A server-supplied window is an instruction.

Distribute load, don't just slow down: Rotating proxy pools spread requests across many IP addresses and user agents to partition cumulative counters and isolate failures.

Model the client as a policer, not a queue: When traffic approaches a limit, the fix isn't to halt the whole pipeline. A request that doesn't meet the token-bucket specification can be delayed, dropped, or marked and deprioritized, without stalling compliant traffic.

Authenticated requests carry separate bucket counters: A rate limit scoped to an API key or session cookie trips independently of IP-based limits. Where the limit is scoped per key rather than per account, spreading work across separately issued credentials keeps each one inside its own limit.

Watch RateLimit headers before you get throttled: Retry-After only fires after a 429 already happened. Quota headers, such as the IETF draft's RateLimit field or the older X-RateLimit-Remaining, let a client see the remaining quota and throttle pre-emptively, before the server ever has to say no.

1. Respect the Retry-After header and build adaptive backoff

Stop guessing when to retry. A server that sends a 429 will often embed the exact wait window in the response headers. Extract that value and obey it.

Parse the Retry-After header for one of its two RFC-defined forms: a delta-seconds integer or an HTTP-date timestamp. Add a small, random jitter of 10 to 15% on top of that window to avoid thundering-herd collisions with other clients that received the same timestamp, then schedule the retry. Retrying before the window ends just earns another 429. When the header is missing, a sign of a less transparent rate limiter, fall back to exponential backoff with a 2-second base interval and a 60-second cap.

The backoff coefficient should only multiply the wait when the server gives no instruction. Resetting base delays on every fresh 200 response keeps a single burst of throttles from permanently crippling throughput. This logic becomes the foundation of every request queue downstream.

Here is that logic as one Python helper. It handles both Retry-After forms, adds jitter on top of the server's window, backs off exponentially only when no header is sent, and reads the remaining quota from whichever RateLimit header style the server uses (fix 3). The usage lines point it at Anakin's URL Scraper, which retries the same way its docs recommend. Set ANAKIN_API_KEY in your environment first. The requests library is the only dependency.

import os
import random
import time
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime

import requests

RETRYABLE = {429, 500, 502, 503}
MAX_ATTEMPTS = 5
BASE_DELAY = 2.0   # seconds
MAX_DELAY = 60.0   # cap for the fallback backoff


def retry_after_seconds(value):
    """Turn a Retry-After header into seconds. Handles both RFC 9110 forms."""
    if not value:
        return None
    value = value.strip()
    if value.isdigit():                      # delta-seconds, e.g. "30"
        return float(value)
    try:                                     # HTTP-date, e.g. "Sun, 25 Oct 2026 10:00:00 GMT"
        when = parsedate_to_datetime(value)
    except (TypeError, ValueError):
        return None
    return max(0.0, (when - datetime.now(timezone.utc)).total_seconds())


def backoff(attempt):
    """Capped exponential backoff with full jitter, used only when the server gives no hint."""
    return random.uniform(0, min(MAX_DELAY, BASE_DELAY * 2 ** attempt))


def remaining_quota(resp):
    """Read remaining quota from IETF-draft or legacy X-RateLimit headers, if the server sends any."""
    rl = resp.headers.get("RateLimit")       # draft-11 style: "default";r=50;t=30
    if rl and ";r=" in rl:
        return int(rl.split(";r=")[1].split(";")[0])
    legacy = resp.headers.get("X-RateLimit-Remaining")
    return int(legacy) if legacy and legacy.isdigit() else None


def request_with_retry(method, url, **kwargs):
    for attempt in range(MAX_ATTEMPTS):
        resp = requests.request(method, url, timeout=30, **kwargs)
        left = remaining_quota(resp)
        if left is not None and left < 5:
            time.sleep(1.0)                  # slow down before the server says no
        if resp.status_code not in RETRYABLE or attempt == MAX_ATTEMPTS - 1:
            return resp
        wait = retry_after_seconds(resp.headers.get("Retry-After"))
        if wait is not None:
            wait *= random.uniform(1.10, 1.15)   # add 10 to 15% so clients don't retry in lockstep
        else:
            wait = backoff(attempt)
        time.sleep(wait)
    return resp


resp = request_with_retry(
    "POST",
    "https://api.anakin.io/v1/url-scraper",
    headers={"X-API-Key": os.environ["ANAKIN_API_KEY"]},
    json={"url": "https://example.com"},
)
resp.raise_for_status()
print(resp.json()["jobId"])

Anakin's API doesn't send X-RateLimit headers today, so against it the quota check simply returns nothing and the Retry-After path does the work. Against targets that do send quota headers, the same helper slows down before the first 429. Anakin's rate limits page documents which statuses to retry and how its own Retry-After behaves.

2. Why do fixed delays fail? Model your client as a token bucket

A fixed delay between requests forces a constant rate. That rate wastes allowance when traffic is light and blows past it the moment a batch bursts. Most servers don't count with a metronome anyway. Their rate limiters run on a burst-tolerant model like Token Bucket(r, Bmax), and a client that doesn't match that model gets throttled first. Matching it client-side keeps requests under the limit without leaving throughput on the floor.

Build this regulator into the request dispatcher:

Instantiate a local bucket. Start a counter at Bmax (depth) and a timer that adds r tokens per second. Cap the counter at Bmax.

Consume tokens before sending. Before transmitting a request, take one token. If the count is zero, suspend the sender.

Define Bmax explicitly. Set Bmax to the burst limit observed or documented by the target. Without documentation, start low at 5 tokens.

Handle non-compliant bursts. When a request arrives and the bucket has no token, don't stall the entire upstream flow. Drop the request into a pending buffer and fire it the instant the next token refills.

Audit fill rate alignment. Compare the configured r to the rate the server signals with 429 responses. Getting throttled while holding a full bucket of unused tokens means r is too high, cut it and re-instantiate.

3. Watch RateLimit headers and throttle before you get throttled

Retry-After is reactive: it only shows up after a server has already rejected a request. A growing share of rate-limited APIs also expose forward-looking quota headers, so a client can see the ceiling coming and slow down before hitting it instead of after.

An IETF draft (revision 11, May 2026) defines two fields, RateLimit-Policy and RateLimit. The policy field states the quota and its window. The RateLimit field reports what's left (its r parameter) and the seconds until reset (t), for example RateLimit: "default";r=50;t=30. Earlier drafts used separate RateLimit-Limit, RateLimit-Remaining and RateLimit-Reset headers, and plenty of production APIs still ship the older X-RateLimit-Remaining and X-RateLimit-Reset convention (a pattern popularized by APIs like GitHub's). Check for all three naming styles rather than assuming one is present.

Read the quota on every response, not just on a 429. Parse whichever remaining-quota header the target sends, on every successful response, not only when a request fails.

Throttle before the header hits zero. When remaining quota drops under roughly 10-15% of the limit, cut the request rate proactively rather than waiting for the first rejection.

Treat the reset window as a scheduling hint. Once remaining quota is nearly exhausted, space out whatever requests are left across the seconds remaining until reset instead of firing them as fast as possible and then stalling.

A rolling-window variant of this idea, tracking the 429 rate itself over a trailing window and halving throughput when it spikes, then easing back up once it's quiet again, is a pattern DataResearchTools' backoff guide also recommends, and it works even against targets that don't expose any quota headers at all.

4. Police the request queue (drop, mark, delay) instead of blocking it

Blocking the entire pipeline when one endpoint throttles is an architectural antipattern. Network engineering handles congestion with policers that selectively delay, drop, or deprioritize packets without stalling compliant traffic. A scraper's request queue should adopt the same model.

Strategy, Action on Non-Compliant Request, and Effect on Pipeline:

A token bucket filter allocates a fixed bandwidth cap to each sender. Unlike fair queuing, it doesn't redistribute idle bandwidth. A queue implementation should avoid pooling all endpoints into one bucket, assigning per-domain or per-IP token buckets that operate in parallel instead. When a specific domain approaches its estimated rate limit, transition from delay to mark for that domain's pending requests. If the server sends a 429 during the marked transmission window, switch to drop and quarantine the endpoint. This graduated policing keeps one aggressive rate limiter from starving twenty other healthy scrapes.

5. Set per-endpoint limits and a retry budget circuit breaker

Two governance gaps show up once a scraper is running against dozens of endpoints instead of one: a single global rate assumption is wrong for most of them, and nothing stops a single broken target from silently burning the whole job's retry budget.

Differentiate limits per endpoint, not per domain. Search, filter, and aggregation endpoints are frequently rate-limited far more aggressively than static content on the same site. ZenRows' guide notes that a single expensive endpoint can trip 429s well before the rest of the domain does, so a scraper that applies one rate limit across an entire domain will either throttle everything to match the strictest endpoint or keep tripping 429s on it. Tag each endpoint with its own bucket and tune independently. Anakin's own rate limits page is a working example: each endpoint is an independent bucket, so POST /v1/wire/task allows 20 requests a minute per user while POST /v1/url-scraper allows 60, and hitting one doesn't touch the other.

Cap the retry budget at the job level, not just per-request. A per-request backoff loop has no concept of the bigger picture: it will happily keep retrying a permanently broken endpoint forever. Set a threshold, for example, abort the job and alert if 429s exceed roughly 15% of total attempts in a run, a pattern also described in DataResearchTools' backoff guide, so one misbehaving endpoint can't eat a batch's entire time budget unnoticed.

Log the breach, don't just recover from it. A retry budget circuit breaker is only useful if it produces a record of what tripped it. Log the endpoint, the observed 429 rate, and the active rate-limit headers at the moment of the abort, so tuning the next run doesn't start from zero.

6. Spread requests across a rotating proxy pool for IP-based limits

The simplest rate limiter counts requests per source IP. Staying within it at volume means spreading traffic across a pool of exit addresses so no single address goes over its share, and resting any address that gets a 429 until its cooldown ends. Rate-limiting restrictions are typically based on a client's IP, but can also target specific users or applications when requests are authenticated or carry a cookie.

Build the pool with two tiers: residential IPs for the most aggressively defended targets, datacenter addresses for high-throughput, low-risk endpoints. Assign IPs on a sticky rotation, reusing the same address for the duration of a logical session when cookie continuity matters, and randomizing uniformly for stateless API calls. When a response returns 429, place that IP into a quarantine map with an expiry time derived from the Retry-After header or a fallback 30-second window. The orchestrator should never dispatch a request to a quarantined address before its slot expires. A managed layer like Anakin's URL Scraper automates this: proxies rotate automatically with fingerprint spoofing applied per target, a country parameter routes requests through any of 207 locations, and a useBrowser toggle renders JavaScript-heavy pages. Anti-bot detection and rate limiting get handled without maintaining the pool by hand.

7. Manage credential state so each per-key limit is respected

Spreading IPs alone isn't enough when limits are scoped to tokens or sessions. Where you hold several legitimately issued credentials, give each one its own share of the work and keep each within its own limit:

Provision token-and-cookie pairs: Pair each session token or API key with its own HTTP cookie jar to prevent stateful interactions from blending across accounts and triggering consistency-based bot heuristics.

Pre-validate every token: Before a batch run, send a lightweight authenticated request against each credential and discard any that return 401 or redirect to a login challenge.

Maintain per-credential counters: During operation, track each credential's request count against the documented limit, and check what that limit is scoped to before you rotate. The Cloudflare API, for example, enforces a global rate limit of 1,200 requests per five-minute period per user, and crossing it blocks all API calls for the next five minutes. That cap counts every API key and token belonging to the same user together, so extra keys on one account buy nothing there. Rotation only adds headroom where the target counts per key.

Automate credential refresh: Intercept 403 or 429 responses to trigger a re-login flow, then atomically update the pool entry so no worker ever pulls a dead token.

That covers requests that carry a token or cookie but are still, structurally, page reads. When the goal is a specific authenticated workflow on a specific site, placing an order, pulling an account's own data, checking availability behind a login, Wire's pre-built catalog actions already carry the correct auth and rate-limit handling for that site, built and tested against its real limits rather than reverse-engineered from response headers. That's narrower than the credential-rotation approach above, but it removes the guesswork entirely for the specific site and action it covers.

8. How do you tell an IP limit from a per-user limit?

Cloudflare's own API is a clean, documented example of the layered model you'll face on many targets. It runs two independent counters: one scoped to the source IP, another to the user or account token. A client that clears one tier can still get a 429 on the other unless both are instrumented. Websites behind Cloudflare set their own thresholds with rate limiting rules, so the numbers below won't transfer, but the two-counter shape and the probing method do:

IP-based burst limit, 200 requests per second: This is Cloudflare's documented per-IP cap for its client API, and it's the layer a fast raw HTTP client hits first. On websites behind Cloudflare, the equivalent block usually shows up not as a plain 429 but as Cloudflare's own rate-limiting page: Error 1015.

Per-user cumulative limit, 1,200 requests per 5 minutes: This limit applies cumulatively regardless of whether the request comes via the dashboard, an API key, or an API token. It catches persistent, low-rate flows that stay under the per-second IP cap.

Fingerprinting methodology: Against an unknown platform, send calibrated probe sequences. First, hit the same endpoint from two different IPs using the same session cookie. Then reverse the test: same IP, different credentials. Whichever tier returns 429 in each test isolates the active limit scope.

Header cataloging: A 429 from a CDN-delivered target often carries diagnostic headers beyond Retry-After. Logging the full header set, especially cf-ray, x-ratelimit-remaining, and custom vendor tokens, helps reverse-engineer the bucket configuration without reading public documentation.

9. Move from raw HTTP to browser sessions where it fits

Every plain HTTP GET request runs the same gamble: the target sees a non-browser TLS handshake before any code reads the first byte of the response. That handshake alone is often enough. Anti-bot systems inspect the TLS fingerprint, decide the client isn't Chrome, and hand back a 429 or a Cloudflare 1020 block before any content loads.

Browser-based scraping addresses the problem at the transport layer. A headless Chromium instance sends a complete browser client hello, identical to what a standard Chrome installation produces. The handshake looks like a standard browser's, so the request is handled like an ordinary browser visit.

Reusing the same browser session pushes the advantage further: log in once inside an isolated profile, and persist the TLS session ID and cookie store.

Reconnecting hours later doesn't start fresh. That one authentication cycle absorbs the proof-of-work cost that a raw HTTP approach would pay on every scrape interval. Each new plain HTTP call starts cold and triggers the same JavaScript challenge again. A resumed browser session can skip it for as long as its clearance cookies stay valid.

Under the hood, this is a headless Chromium instance controlled over the DevTools Protocol. Anakin's Browser API handles navigation, clicking, form fills, and extraction while storing encrypted cookies and local storage across disconnects. The target here is a specific defense model: Cloudflare blocks automation at three independent layers, TLS fingerprinting during handshake, IP reputation during routing, and JavaScript challenges after page load.

A raw HTTP client doesn't address any of those three layers. A persistent browser session with native TLS fingerprinting and deterministic challenge-solving hits all three at once.

Not every target needs a full browser to clear this bar. For raw HTTP paths where spinning up Chromium is overkill, ZenRows' guide on handling 429s recommends rotating realistic, internally-consistent header sets (User-Agent, Accept, Accept-Language) rather than randomizing individual headers per request, since a browser never changes its User-Agent mid-session and mismatched header combinations are themselves a fingerprinting signal. Anakin's Browser API handles the same consistency problem at the network level: pass ?country=GB and the session's timezone, locale and Accept-Language all match the UK residential proxy it routes through, so the headers and the IP tell the same story.

10. Or skip the plumbing: what Anakin handles for you

Building adaptive backoff, proxy pools, and credential rings from scratch is a months-long side project. Anakin.io consolidates these controls into a single execution interface, so an infrastructure team stops managing rate-limiting plumbing and starts consuming structured data. URL Scraper is the default entry point for most of this: its architecture models target endpoints behind independent, concurrency-bounded worker pools that absorb rate-limit signals without collapsing the whole pipeline. When a target issues a 429, the associated worker throttles in isolation while other domains keep running at full speed.

Every authenticated session inside the platform is encrypted at rest with AES-256-GCM and scoped to the account. Credential values never appear in API responses or log lines, which satisfies the audit requirement that agents and downstream collaborators never touch raw secrets. Sessions connect over the DevTools Protocol and are ready in roughly 180ms, with encrypted cookies and local storage persisting across disconnects instead of writing plaintext passwords to disk.

The caching layer cuts redundant target-server hits. Repeat requests for the same URL within 24 hours return instantly from cache and cost zero credits. They still count against Anakin's own API rate limits, but they never reach the target site, so a dashboard re-reading the same product pages throughout the day sends one request to the target per 24-hour cache window, not one per read. On repeated reads, that's 429 avoidance you don't have to tune.

Agent setup is one command in Claude Code: claude mcp add --transport http anakin https://mcp.anakin.io/mcp connects to the hosted MCP server, which authorizes with OAuth 2.1, so there's no API key to paste. The @anakin-io/mcp package covers Cursor, Windsurf, VS Code and other MCP clients. The free tier provisions 300 credits on signup with no credit card required, and credits are only deducted on success: failed scrapes, timeouts, and cache hits cost nothing. For high-volume pipelines, the Scale plan provides 100,000 credits a month at a 50-concurrent limit with priority support. Anakin started in 2021 as a Y Combinator company (S21) and runs infrastructure built for the dual constraints of aggressive anti-bot defenses and lawful, auditable data access.

Which fix should you start with?

Look at when the 429 arrives and what comes back with it. Each symptom points to one fix:

• 429 on the very first raw HTTP request, before any content loads: a transport-layer block. Start with fix 9.
• 429 with a Retry-After header: obey the window and add jitter. Start with fix 1.
• 429 after a burst, even though the average rate looks fine: the bucket depth is the problem. Start with fix 2.
• Quota headers on every response: throttle before they hit zero. Start with fix 3.
• One endpoint failing while the rest of the site loads: give it its own bucket and a retry budget. Start with fix 5.
• No idea whether the limit is per IP or per credential: run the two probes in fix 8 before you spread load with fixes 6 and 7.

Pace like a bucket, not a metronome

A sleep() call paces requests like a metronome, and that's why it fails. Stable, high-volume scraping in 2026 means reading what the server tells you and matching how it counts. Start by parsing Retry-After and shaping the client with a local token bucket that matches the server's TB(r, Bmax) model. Then spread load across a rotating proxy pool with per-IP quarantine, spread work across credentials only where limits are scoped per key, and police the queue with drop-mark-delay semantics so one throttled endpoint doesn't stall the rest.

The fix order above is the build order: read the server first, then pace, then spread load. Whatever the symptom, keep the header monitoring and per-endpoint retry budget from fixes 3 and 5 running, and most of what's left is watching the logs. Anakin's URL Scraper and Browser API take the proxy, fingerprint and session layers off your plate, so the time goes into the part only you know: your targets and their limits.

Frequently Asked Questions

What exactly is an HTTP 429 status code and why do websites return it?

HTTP 429 Too Many Requests means the server determined too many requests arrived in a given amount of time. Sites return it to protect their infrastructure from overload, prevent resource exhaustion, and enforce rate-limiting policies that distinguish legitimate traffic from abusive automation.

Why do delays between requests sometimes fail to prevent 429 errors?

Uniform delays fail because modern rate limiters often use token bucket algorithms that specify both a fill rate and a burst depth. A fixed 2-second delay sends 0.5 requests per second forever. If the bucket refills more slowly than that, the delay drains it no matter how evenly the requests are spaced, and if it refills faster, the delay wastes the burst allowance the bucket was built to give.

What rate-limiting models and algorithms do modern websites use that make simple delays insufficient?

Token bucket dominates, defined as TB(r, Bmax) with a token fill rate of r tokens per second and a bucket capacity Bmax. It permits a burst up to Bmax tokens, then enforces a strict average rate of r. Leaky bucket and sliding window algorithms are also common.

How do you identify whether a scraper is hitting IP-based, session-based, or endpoint-specific rate limits?

Run two controlled probes: the same request from two IPs with the same session cookie, then the same IP with different credentials. Whichever test returns 429 shows the active limit scope. Fix 8 walks through the method.

What code patterns and HTTP header handling handle 429 errors gracefully?

Read Retry-After first, in its delta-seconds or HTTP-date form, add 10 to 15% jitter and retry after that window. Fall back to capped exponential backoff only when the header is absent. Fix 1 includes a Python example.

What architectural approaches beyond delays do professional scraping services use to avoid 429 errors at scale?

Professional infrastructure combines rotating residential and datacenter proxy pools with per-IP quarantine maps, per-credential limits across separately issued API tokens, token bucket queue policing with drop-mark-delay semantics, and browser-based session reuse that clears a JavaScript challenge once per session instead of on every request. Each of these is a system-level control, not a script-level tweak.

What's the difference between the Retry-After header and RateLimit headers?

Retry-After is reactive: it only appears on a 429 or 503 response, after the limit has already been hit. Quota headers, the IETF draft's RateLimit and RateLimit-Policy fields or the older X-RateLimit-Remaining and X-RateLimit-Reset many APIs still use, are forward-looking and can appear on every response, letting a client see the remaining quota and throttle down before it ever gets a 429 at all.

Get started with 300 free credits, no card required, on Anakin.io, one API key across URL Scraper, Wire, and Browser API. See Anakin's URL Scraper for rate-limit-aware page reads, or Wire's catalog for authenticated actions that skip the guesswork entirely.