A useful API limit must do more than reject requests after a counter reaches a number. It needs to define whose traffic is being limited, which bursts are legitimate, how the rule behaves across application replicas, and what a client should do after receiving a rejection.
Rate limiting is an admission-control policy. It protects a service and its dependencies from a client, tenant, integration or workload that is consuming more than its agreed share. A poorly chosen policy can be just as disruptive as having no policy: it may block legitimate bursts, multiply quotas across replicas, or cause every client to retry at the same moment.
This guide uses a token bucket as a practical starting point. It focuses on the design decisions around the algorithm rather than treating a calculator output as a production configuration. The accompanying Token Bucket Rate Limit Calculator models burst allowance, refill capacity and the effect of independent per-instance buckets.
1. Define the boundary before choosing a limit
Write down exactly what the limit protects and who owns the quota.
| Limit scope | Typical key | Design question |
|---|---|---|
| Per user | Authenticated user ID | Can one user create multiple identities to bypass the quota? |
| Per API key or client | Stable credential ID | Are credentials rotated or shared by several applications? |
| Per tenant | Tenant or account ID | Should a large tenant receive a different contracted allowance? |
| Per endpoint | Route plus identity | Do expensive operations need a lower limit than inexpensive reads? |
| Global service | Shared service key | Is the service protecting a downstream dependency with a finite total budget? |
Avoid using an untrusted request header as the identity. If the service is behind a reverse proxy, accept forwarded client-address headers only from known proxy hops and configure the trusted-proxy boundary deliberately. Otherwise a caller may supply a different apparent address and create new buckets.
IP-based limits can still be useful for unauthenticated endpoints, but they are not a substitute for account-level policy. NAT, corporate gateways and mobile carrier networks can put many legitimate users behind one address. Combine identity, route and abuse signals according to the risk of the endpoint.
2. Understand what the token bucket actually allows
A token bucket has two main parameters:
- Capacity (B): the maximum number of requests that can be admitted as a burst when the bucket is full.
- Refill rate (r): the number of tokens added per second.
Each accepted request consumes one token. Tokens accumulate up to the capacity; extra refill is discarded while the bucket is full. A request arriving with no available token is rejected or queued, depending on the service contract.
For one bucket, an upper bound on the number of requests that can be admitted during a constant-rate interval is:
admitted ≤ min(incoming requests, initial tokens + refill rate × interval)
This is a planning bound, not a replacement for a discrete request simulation. It assumes one token per request, a constant arrival rate and a refill rate that is maintained for the whole interval.
Worked example: a five-second burst
Assume an API key has:
- Bucket capacity: 120 requests.
- Refill rate: 20 requests per second.
- Initial bucket fill: 120 tokens.
- Incoming traffic: 50 requests per second.
- Burst duration: 5 seconds.
The interval contains about 250 incoming requests. The bucket starts with 120 tokens and can add another 100 during the five seconds:
- Available token supply: 120 + (20 × 5) = 220.
- Estimated admitted requests: 220.
- Estimated rejected requests: 250 − 220 = 30.
- The initial burst is exhausted after approximately 120 ÷ (50 − 20) = 4 seconds.
The rate limiter absorbs the first part of the burst, then admits traffic at roughly the refill rate while rejecting the excess. If the arrival rate remains above the refill rate, the same pressure continues after the initial tokens are gone.
3. Decide whether the bucket is local or shared
A frequent distributed-systems mistake is configuring the same full token bucket independently in every application replica and assuming the configured rate is a global limit.
Suppose four replicas each have a capacity of 120 and refill at 20 requests per second. If traffic is distributed evenly, the aggregate model is:
- Aggregate burst capacity: 4 × 120 = 480 requests.
- Aggregate refill rate: 4 × 20 = 80 requests per second.
The effective service allowance is therefore four times the per-instance setting. This may be intentional when each replica protects its own local resource. It is usually not what a product owner means by a global tenant quota.
There are three common approaches:
- Local buckets: low coordination overhead, but each replica has an independent allowance. Autoscaling and uneven load balancing change the effective aggregate limit.
- Shared state: all replicas consult one authoritative limiter, often backed by Redis. This gives a more consistent global decision but adds a network dependency to the request path.
- Partitioned quotas: allocate a bounded share to each region or instance. This can reduce coordination, but the allocation and rebalancing rules must be explicit.
The calculator lets you model independent buckets as a group. Set the bucket count to one for a single shared bucket. When modeling several local buckets, its aggregate result assumes traffic is evenly distributed; a hot replica or sticky routing can cause one bucket to reject earlier.
For a Redis-backed limiter, the read-refill-decide-update operation must be atomic. A Lua script or another atomic server-side operation prevents concurrent requests from reading the same token balance and both spending the last token. Use a consistent time source for refill calculations across application instances. Redis documents token-bucket implementations and the trade-offs between fixed windows, sliding windows and token buckets in its rate-limiter guide.
4. Choose the algorithm for the traffic shape
Token buckets are useful when a client may send a short burst but should settle to a sustained rate. They are not the only option.
| Algorithm | Behaviour | Useful when |
|---|---|---|
| Fixed window counter | Counts requests in a fixed time interval | A simple quota is sufficient and boundary bursts are acceptable |
| Sliding window log | Tracks individual request timestamps | A more exact rolling-window limit justifies the extra state |
| Sliding window counter | Estimates a rolling window using adjacent counters | A smoother boundary is needed with less state than a full log |
| Token bucket | Allows a bounded burst and refills continuously | Short bursts are valid but sustained demand must be bounded |
| Leaky bucket / queue | Releases work at a controlled pace | Smoothing is more important than immediate admission |
Do not select an algorithm because it is familiar. Write down the expected burst, the sustained quota, the fairness requirement and the cost of coordinating state. Then test the policy with realistic arrival patterns.
5. Make rejection useful to the client
For an HTTP API, a client-specific quota rejection is commonly returned as 429 Too Many Requests. RFC 6585 defines the status and permits a Retry-After header. RFC 9110 describes the header's two forms: a delay in seconds or an HTTP date.
A response might look like:
HTTP/1.1 429 Too Many Requests
Retry-After: 2
Cache-Control: no-store
Content-Type: application/problem+jsonThe delay should reflect the policy's next useful retry opportunity, not be an arbitrary constant. Clients should wait at least as long as the server instructs, then use bounded retries and jitter where appropriate. See the HTTP 429 guidance in RFC 6585 and Retry-After in RFC 9110.
A 429 response is not the right answer to every overload condition. It usually describes a caller exceeding a quota. If the entire service is unhealthy or temporarily unable to accept work, a service-level response such as 503 may be more accurate. Keep the response contract aligned with the failure being reported.
Rate limiting is also different from concurrency limiting. A request-rate policy controls how frequently work is admitted; it does not stop a small number of very slow requests from occupying every worker. Expensive or long-running operations may also need a semaphore, queue limit, bulkhead or workload-specific concurrency cap.
6. Make the policy safe to operate
A production policy needs more than a limit and a counter.
- Atomic decisions: avoid race conditions when several requests arrive for the same key.
- Expiry and cleanup: remove inactive bucket state so key cardinality does not grow without bound.
- Fairness: decide how shared tenant quotas, endpoint weights and priority traffic interact.
- Failure mode: document whether a Redis outage fails open, fails closed or uses a bounded local fallback. The right choice depends on the protected resource and abuse risk.
- Autoscaling: account for independent local buckets when replicas are added or removed. A deployment should not silently multiply a global quota.
- Observability: measure allowed and rejected requests, remaining tokens, 429 rates by route and tenant, limiter latency, backend errors and key cardinality. Avoid logging raw API keys or personal identifiers.
- Load testing: test steady traffic, synchronized bursts, uneven routing, hot tenants, limiter-backend latency and failover.
A rate-limit change should have an owner, a rollout plan and a rollback condition. For broader traffic assumptions, use the network infrastructure capacity-planning guide and the System Design Capacity Calculator. For platform changes, connect the policy to the AKS production-readiness guide and its upgrade preflight checklist.
7. Connect rate limits to the rest of the architecture
A rate limiter is one part of a larger reliability boundary:
- Use the Redis versus Kafka decision guide when deciding whether quota state belongs in a low-latency shared store or whether the event stream is only needed for audit and asynchronous analysis.
- For asynchronous work, review the Kafka notification architecture and the consumer lag drain-time calculator.
- If a rejected request is retried, apply the retry backoff and budget guide and its retry calculator. A retry policy that ignores 429 and Retry-After can turn a protective limit into a repeated burst.
- Define the service-level objective and the operational response to sustained throttling with the SLO error-budget guide and SLO Error Budget Calculator.
- Protect the origin and trusted proxy boundary with the Cloudflare and Nginx origin-security guide.
The practical goal is not to maximize the number of requests rejected. It is to make admission predictable: legitimate clients get a fair share, bursts have a documented allowance, dependencies remain protected, and clients know how to recover without amplifying an incident.