Fault-Tolerant Distributed Rate Limiter

Most rate limiters only work if you have one server running. The moment you put four servers behind a load balancer, each one starts enforcing its own separate limit — a client that keeps landing on the same server can blow through the real limit by a factor of however many servers you're running. And the fix everyone reaches for, a shared Redis counter, has its own failure mode: when Redis goes down, the limiter either fails open (nobody gets rate-limited) or fails closed (everybody gets blocked), including the users who did nothing wrong.

I spent the last two days building a rate limiter that avoids both problems: four app instances behind nginx sharing one Redis for the common case, and a local in-memory fallback with its own reconciliation step for when Redis isn't there. This is project 1 of 45 in a backend-engineering series, and it's the first one that landed under a section I'm calling Coordination & Consistency — problems where the interesting part isn't the happy path, it's keeping multiple things that don't fully trust each other in agreement.

Repo: github.com/SafalBhandari12/Fault-Tolerant-Distributed-Rate-Limiter

The problem with "just add Redis"

A single-process rate limiter is trivial: keep a counter in memory, check it, increment it. It falls apart the instant you scale horizontally, because "memory" now means four different memories that don't talk to each other.

One shared limit enforced by four independent servers, versus each server keeping its own counter Without a shared store, a limit of 5 per server becomes a limit of 5 × N servers in practice.

Point every server at the same Redis instance and the counter problem goes away — but you've traded it for a new one. Redis is now a single point of failure sitting directly on your request path. If it hiccups, every rate-limit check either throws (fail closed, you just took your whole API down) or silently returns "allowed" (fail open, you just turned your rate limiter off during the exact moment — likely a traffic spike or partial outage — when you needed it most).

Two counters, both atomic

I implemented both of the standard rate-limiting algorithms, and both live entirely inside a Redis Lua script rather than round-tripping read-then-write from application code:

Token bucket — each key has a token count that refills at a fixed rate and drains per request:

local tokens = tonumber(data[1]) or capacity
local lastRefill = tonumber(data[2]) or now
local elapsed = math.max(0, now - lastRefill)
tokens = math.min(capacity, tokens + elapsed * refillRate)
local allowed = tokens >= cost
if allowed then
    tokens = tokens - cost
end
redis.call('HMSET', KEYS[1], 'tokens', tokens, 'lastRefill', now)
return allowed and 1 or 0

Sliding window — a sorted set of request timestamps per key, trimmed to the window on every check:

redis.call('ZREMRANGEBYSCORE', key, 0, now - window)
local count = redis.call('ZCARD', key)
if count < limit then
    redis.call('ZADD', key, now, requestId)
    return 1
else
    return 0
end

The reason both live in Lua and not in TypeScript: Redis runs a Lua script as a single atomic operation. If this were "GET the count, check it in Node, SET the new count," two servers could both read the same count, both decide the request is allowed, and both write back — a classic check-then-act race. Wrapping the read-modify-write in one script means Redis serializes it for you; every one of the four app instances is hitting the exact same atomic operation, so there's no window where two servers can both think they got the last token.

Detecting that Redis is gone

Fail-open and fail-closed are both wrong because they're both blind: they don't distinguish "Redis is fine" from "Redis is unreachable." What you actually want is a third state — keep rate-limiting, just do it locally until Redis comes back. That's what a circuit breaker gives you.

Circuit breaker state machine: CLOSED checks Redis directly, three consecutive failures flips to OPEN where servers use a local fallback, one successful health check flips back to CLOSED Every server tracks this state independently — no coordination is needed to agree that Redis is down.

class CircuitBreakerClass {
  state: "CLOSED" | "OPEN" = "CLOSED";
  failureCount = 0;
 
  success() {
    this.failureCount = 0;
    this.state = "CLOSED";
  }
 
  failure() {
    this.failureCount++;
    if (this.failureCount >= 3) this.state = "OPEN";
  }
}

Each rate-limit call is wrapped in a try/catch: if the Redis eval throws, that counts as a failure, and once three failures stack up the breaker opens. A separate 1-second interval pings Redis independently and flips the breaker back to CLOSED the moment a connection succeeds. Three failures instead of one keeps a single dropped packet from tripping the whole thing; a 1-second poll keeps the blind window short once it does trip.

While the breaker is open, allowRequest() skips Redis entirely and asks a local LeaseManagerClass instead — an in-memory remaining-count per key, refreshed periodically from the last known Redis value divided across the four instances, so a full Redis outage still rate-limits people, just against a slightly stale, locally-approximated number instead of nothing at all.

The part that's actually hard: getting back in sync

Falling back to local memory is the easy half. The hard half — the part I haven't seen covered in the guides I read before starting this — is what happens when Redis comes back. During the outage, four servers independently granted requests against their own local leases. Redis has no idea any of that happened; its counters are frozen at whatever they were the instant it went down. If you just resume hitting Redis normally, you silently erase every request that got through during the outage — Redis will let a burst of yet more traffic through because, as far as it knows, nothing happened in the meantime.

Four-step reconciliation flow: outage grants get logged locally, Redis recovers, the local log is replayed into Redis, then cleared Skip this step and an outage either erases the usage that happened during it, or double-counts it once Redis is back.

So every local grant gets logged — {timestamp, cost} — while the breaker is open. The moment the health-check interval sees Redis succeed and notices the breaker had been open, it runs a reconciliation pass before resuming normal traffic:

async function reconcileLeases() {
  for (const lease of tokenBucket.getAll()) {
    if (lease.log.length === 0) continue;
    const totalCost = lease.log.reduce((sum, e) => sum + e.cost, 0);
 
    await redis.eval(`
      local tokens = tonumber(redis.call('HGET', KEYS[1], 'tokens')) or 0
      redis.call('HSET', KEYS[1], 'tokens', math.max(0, tokens - tonumber(ARGV[1])))
    `, { keys: [lease.key], arguments: [String(totalCost)] });
 
    tokenBucket.clearLog(lease.key);
  }
 
  for (const lease of slidingWindow.getAll()) {
    for (const entry of lease.log) {
      await redis.zAdd(`rate:users:${lease.key}`, {
        score: entry.timestamp,
        value: randomUUID(),
      });
    }
    slidingWindow.clearLog(lease.key);
  }
}

The two algorithms need genuinely different replay strategies, and the reason why is the detail worth sitting with:

Sliding window replays cleanly, one entry at a time — each logged request just becomes a ZADD with its original timestamp. The sorted set doesn't care what order entries arrive in, only what timestamps they carry, so replaying the whole log entry-by-entry is safe.
Token bucket can't be replayed the same way. By the time reconciliation runs, Redis's tokens value has already refilled for however long the outage lasted — replaying each local grant as its own decrement would apply that already-refilled number to every single step, effectively double-crediting the refill N times. Instead, the fix sums the total cost consumed locally during the outage and subtracts it once, against Redis's current (already-refilled) token count.

Once a lease's log is replayed, it's cleared — otherwise the next outage would replay the same entries again on top of new ones, double-counting usage a second time.

What running it looks like

The whole thing runs as four Node/Express instances (app1–app4) behind an nginx round-robin upstream, all pointed at one Redis container, orchestrated with docker-compose. Killing the Redis container mid-load-test is the actual test: requests keep getting 429'd correctly through the outage via each server's local lease, and bringing Redis back doesn't reset anyone's count to zero or let a burst of double-quota traffic through.

The genuinely hard part wasn't the Lua scripts or the circuit breaker — both are well-trodden patterns. It was realizing that "fall back to local state" is only half a solution; the other half is deciding, precisely, how local state gets folded back into the source of truth without losing information or counting it twice. That reconciliation step is the piece most rate-limiter writeups skip entirely, probably because it's the piece that doesn't show up until you actually kill Redis and watch what breaks.

Next in Coordination & Consistency: more projects where the failure mode, not the happy path, is the actual design problem.