Metis

Field note / architecture

publishedupdated 2026-05-01#system-design#rate-limiter#redis#distributed-systems#interview

System Design: Rate Limiter

Design a distributed rate limiter — token bucket vs sliding window, Redis implementation, and where to place it in your architecture.

Overview

A rate limiter controls how many requests a client can make in a time window. It protects services from abuse, prevents resource exhaustion, and enforces fair usage in multi-tenant systems.

Interview scope: Design a rate limiter that:

  • Handles 10,000 RPS across 10M users
  • Works across multiple API gateway nodes (distributed)
  • Allows different limits per user tier (free: 100 RPM, paid: 1000 RPM)
  • Returns 429 Too Many Requests with Retry-After header

Architecture

 Client Requests
      │
      ▼
 ┌────────────────────────────────────────────────┐
 │              API Gateway Cluster                │
 │                                                 │
 │  Node 1 ──▶ Rate Limit Check ──▶ Forward/429   │
 │  Node 2 ──▶ Rate Limit Check ──▶ Forward/429   │
 │  Node 3 ──▶ Rate Limit Check ──▶ Forward/429   │
 │                   │                             │
 │                   │ Lua script (atomic)         │
 └───────────────────│─────────────────────────────┘
                     │
                     ▼
 ┌──────────────────────────────────────────────────┐
 │              Redis Cluster                        │
 │                                                   │
 │  Key: rate:{userId}:{window}                     │
 │  TTL: window size (60s)                          │
 │                                                   │
 │  ┌──────────────┐   ┌──────────────┐             │
 │  │ Shard 1      │   │ Shard 2      │             │
 │  │ user:1..3M   │   │ user:3M..6M  │  ...        │
 │  └──────────────┘   └──────────────┘             │
 └──────────────────────────────────────────────────┘
                     │
                     ▼
 ┌──────────────────────────────────────────────────┐
 │           Downstream Services                     │
 │   (only reached if rate limit not exceeded)       │
 └──────────────────────────────────────────────────┘

 Response Headers (on every request):
 ┌─────────────────────────────────────────────┐
 │  X-RateLimit-Limit: 100                     │
 │  X-RateLimit-Remaining: 43                  │
 │  X-RateLimit-Reset: 1714600800              │
 │  Retry-After: 47  (only on 429)             │
 └─────────────────────────────────────────────┘

Algorithms

Token Bucket (Recommended for most cases)

Bucket has capacity C tokens.
Tokens refill at rate R per second.
Each request consumes 1 token.
If bucket is empty → reject.

Pros: allows short bursts up to C
Cons: burst can hammer downstream if C is large

Sliding Window Counter (Interview favorite — explain this one)

Window: last 60 seconds, split into 1-second slots.
Count requests in each slot.
Sum slots within the window.
If sum > limit → reject.

  time: 0  1  2  3  4  5  ...  58 59 60
        ─  ─  ─  ─  ─  ─       ─  ─  ─
count:  2  0  3  1  0  4  ...   2  1  ← current slot

Current window sum = sum(slots[now-60..now]) = 13

More accurate than fixed window, less memory than per-request log.

Fixed Window (Simple but has burst problem)

Window: current minute (e.g., 12:00:00 – 12:01:00)
Count resets at window boundary.

Problem: user can send 100 at 12:00:59 and 100 at 12:01:00
→ 200 requests in 2 seconds, despite 100/min limit.

Technical Implementation

Redis Lua Script — Atomic Sliding Window

-- rate_limit.lua
-- KEYS[1] = rate:{userId}:{windowStart}
-- ARGV[1] = current timestamp (ms)
-- ARGV[2] = window size (ms)
-- ARGV[3] = limit

local key = KEYS[1]
local now = tonumber(ARGV[1])
local window = tonumber(ARGV[2])
local limit = tonumber(ARGV[3])

-- Remove counts older than window
redis.call('ZREMRANGEBYSCORE', key, 0, now - window)

-- Count current window
local current = redis.call('ZCARD', key)

if current >= limit then
  return {0, current, limit}  -- rejected
end

-- Add current request
redis.call('ZADD', key, now, now .. '-' .. math.random())
redis.call('PEXPIRE', key, window)

return {1, current + 1, limit}  -- allowed

// src/middleware/rate-limiter.ts
import { createClient } from 'redis';
import fs from 'fs';

const redis = createClient({ url: process.env.REDIS_URL });
const script = fs.readFileSync('rate_limit.lua', 'utf-8');

export async function checkRateLimit(
  userId: string,
  tier: 'free' | 'paid'
): Promise<{ allowed: boolean; remaining: number; resetMs: number }> {
  const limits = { free: 100, paid: 1000 };
  const windowMs = 60_000;  // 1 minute

  const [allowed, current, limit] = await redis.eval(
    script,
    { keys: [`rate:${userId}`], arguments: [Date.now().toString(), windowMs.toString(), limits[tier].toString()] }
  ) as [number, number, number];

  return {
    allowed: allowed === 1,
    remaining: Math.max(0, limit - current),
    resetMs: windowMs,
  };
}

// Express middleware
export const rateLimitMiddleware = async (req: Request, res: Response, next: NextFunction) => {
  const userId = req.user?.id ?? req.ip;
  const tier = req.user?.tier ?? 'free';

  const { allowed, remaining, resetMs } = await checkRateLimit(userId, tier);

  res.setHeader('X-RateLimit-Limit', tier === 'paid' ? 1000 : 100);
  res.setHeader('X-RateLimit-Remaining', remaining);
  res.setHeader('X-RateLimit-Reset', Math.floor((Date.now() + resetMs) / 1000));

  if (!allowed) {
    res.setHeader('Retry-After', Math.ceil(resetMs / 1000));
    return res.status(429).json({ error: 'Rate limit exceeded' });
  }

  next();
};

Interview Preparation

Q: How do you handle the distributed case — multiple API nodes?

All nodes share Redis. The Lua script runs atomically on Redis — no race conditions. Redis is the single source of truth for counts. If Redis is down, fail open (allow) or fail closed (reject) — configurable per SLA.

Q: What if Redis goes down?

Strategy 1: Fail open — allow all requests. Abuse gets through temporarily, but service stays up. Right for most APIs. Strategy 2: Local fallback — each node tracks counts locally. Approximate, allows 10× the limit if 10 nodes. Wrong for strict billing. Strategy 3: Circuit breaker — if Redis is down > 5s, return 503 to the client ("service unavailable"). Right for billing/compliance.

Q: Fixed window vs sliding window — what's the bug in fixed window?

Burst at window boundary. A user sends 100 requests at 11:59:59 and 100 at 12:00:01. Both windows count 100 (under limit), but the service got 200 in 2 seconds. Sliding window prevents this by counting over a true rolling period.

Q: How do you rate limit by IP when behind a load balancer?

Use X-Forwarded-For header (last IP before your server, or the first in the chain depending on trust). Configure your load balancer to set this reliably. Be careful — headers are easily spoofed by clients not behind your LB.

Q: How do you implement different rate limits for different endpoints?

Composite key: rate:{userId}:{endpoint}:{window}. /api/search gets 50 RPM, /api/upload gets 5 RPM. Endpoint limits defined in config, looked up in the middleware before the Redis call.

Learning Resources