Retries are useful when a failure is temporary. They are dangerous when every caller retries at the same moment, a request is repeated after an unknown commit, or the retry schedule quietly outlives the user-facing deadline. A reliable policy needs to bound both the number of attempts and the time those attempts are allowed to consume.
This guide focuses on practical retry decisions for HTTP APIs, background jobs and event consumers. It complements the existing Kafka event-driven notification architecture and Redis vs Kafka decision guide: the goal is not to retry everything, but to recover from the right failures without amplifying an incident.
1. Decide whether the operation is safe to retry
Start by classifying the failure, not by choosing a delay.
| Failure | Typical response | Why |
|---|---|---|
| Connection reset before a response | Often retryable within the deadline | The outcome may be unknown; the operation still needs idempotency protection. |
| HTTP 429 or an explicit throttling response | Back off and respect a valid Retry-After value | Immediate retries increase pressure on an already constrained service. |
| HTTP 502, 503 or 504 | Often retryable for safe operations | The upstream may be temporarily unavailable, but repeated attempts must remain bounded. |
| Validation or authorization failure | Do not retry unchanged input | The same request is likely to fail again and adds no useful recovery opportunity. |
| A business conflict | Follow the domain's conflict policy | Repeating a stale command may be incorrect even if the transport is healthy. |
| A timeout after a write may have committed | Retry only with a safe idempotency strategy | A timeout does not prove that the server rolled back the operation. |
These are starting points, not universal HTTP rules. The endpoint contract and business semantics decide what is safe. For example, a payment or bid submission should not be repeated merely because the caller did not receive a response.
For state-changing operations, use an idempotency key or a durable deduplication mechanism where the service supports it. The server must persist enough information to return the original outcome for a repeated logical request. A process-local cache is not a sufficient guarantee when requests can reach different replicas.
2. Bound the exponential backoff
A common schedule increases the delay after each failed attempt:
delay = minimum(maximum delay, base delay × multiplier to the retry index)
Suppose a client starts with a 200 ms delay, doubles it after each retry and caps it at 2 seconds. The uncapped sequence is 200, 400, 800, 1,600, 3,200 ms; the cap turns the fifth delay into 2,000 ms.
A cap matters because an uncapped exponential can grow beyond the useful lifetime of a request or job. The cap does not mean the client should keep retrying forever. Maximum attempts, an overall deadline and an operational retry budget are separate controls.
Use measured service behaviour to choose the base delay and cap. A latency-sensitive interactive API may need a much shorter budget than a durable background workflow. Avoid copying one retry schedule across every dependency without checking their timeouts and rate limits.
3. Add jitter to avoid synchronized retries
If hundreds of clients receive the same failure and all wait exactly 400 ms, they can send another burst together. This synchronized pattern is one way a small outage becomes a retry storm.
Jitter randomizes the delay within a defined range. Common strategies include:
- No jitter: use the capped delay exactly. Easy to predict, but callers can synchronize.
- Equal jitter: choose a delay between half the capped delay and the full delay.
- Full jitter: choose a delay between zero and the capped delay.
For a capped delay of 400 ms, equal jitter has a range of 200–400 ms and a midpoint of 300 ms. Full jitter has a range of 0–400 ms and a midpoint of 200 ms. The midpoint is a planning estimate, not a promise about a particular random draw.
The Retry Backoff & Budget Calculator lays out the delay range for each retry and compares the schedule with a request deadline. It intentionally calculates ranges rather than generating random delays, so the result remains repeatable and testable.
4. Budget the entire operation, not just the sleep time
A retry policy consumes more than backoff delay. It also consumes connection setup, request processing, response handling and possibly queue time. If each attempt can run for 800 ms, three attempts can consume up to 2,400 ms of attempt time before any backoff is counted.
For a simple timeout-bound planning model:
worst-case duration = attempt count × per-attempt timeout + maximum backoff wait
Using three attempts, an 800 ms timeout and delays of 200 ms and 400 ms gives a worst-case envelope of 3 × 800 + 200 + 400 = 3,000 ms. With equal jitter, the expected wait in this simplified model is 150 + 300 = 450 ms, but the conservative envelope still uses the full 600 ms of possible wait.
This estimate assumes every attempt consumes its full timeout. Real attempts may finish sooner, and network or queue overhead may add time not represented by the model. Use the estimate to expose a bad policy, then validate it against traces and actual client behaviour.
The overall deadline must include all attempts and delays. If the caller has a 2-second deadline, a retry policy with a 3-second worst-case envelope is not made safe by the fact that its backoff cap is small. Stop when the remaining deadline cannot accommodate another meaningful attempt.
5. Use retry budgets and backpressure
A maximum-attempt count limits one operation. A retry budget limits the aggregate extra work a system creates while it is unhealthy. For example, a service might allow only a small proportion of requests to be retries over a measurement window. The exact policy depends on traffic patterns and failure cost; it should be observable and enforced rather than left as an informal guideline.
When a dependency is overloaded, consider these controls together:
- Respect server-provided Retry-After guidance when it is valid and compatible with the remaining deadline.
- Limit concurrent in-flight requests and apply backpressure to producers.
- Use a circuit breaker or load-shedding policy when repeated calls are unlikely to succeed.
- Avoid retrying at every layer. Three attempts in a browser, three in an API gateway and three in a service can multiply into many downstream calls.
- Add bounded queues and dead-letter or manual-review paths for work that cannot be completed safely.
- Emit metrics for attempts per logical operation, retries by reason, exhausted budgets, timeout rates and final outcomes.
A retry budget is different from an SLO error budget. The SLO error-budget guide expresses allowed unreliability over a service window. A retry budget limits extra load created by recovery attempts. Both can inform operational decisions, but they measure different things.
6. Be deliberate with message consumers
A Kafka consumer may retry a transient downstream failure, pause a partition, or route a repeatedly failing message through a dead-letter workflow. Those choices affect ordering and lag. Sleeping inside a consumer thread can reduce useful throughput; immediately reprocessing a poison message can create a hot loop.
The right pattern depends on ordering requirements, the client's retry mechanism and whether later records can safely proceed. Record the original event identifier, make downstream side effects idempotent, use bounded retry attempts and make exhausted retries visible to operators. If a message is retried out of order, document the business consequence rather than assuming ordering is preserved.
The Kafka Consumer Lag Drain-Time Calculator can help estimate whether a consumer group can catch up while new records arrive. If a retry policy reduces sustained processing throughput, re-run the capacity estimate using measured successful processing rates, including the cost of retries.
7. Test the failure policy, not just the happy path
A useful retry test suite should cover at least:
- A transient failure followed by success.
- A permanent failure that must not be retried.
- A timeout where the server may already have committed the operation.
- Repeated delivery with the same idempotency key.
- Retry-After longer than the remaining deadline.
- Backoff capped at the configured maximum.
- Jitter ranges at their minimum and maximum boundaries.
- A deadline too short for even one attempt.
- Retry exhaustion and the resulting alert, dead-letter or user-visible outcome.
Test with injected clocks and deterministic random sources when verifying a real retry implementation. Avoid tests that rely on wall-clock sleeps or a particular random sample; they tend to be slow and flaky.
For production changes, agree on stop conditions before the rollout. The AKS upgrade preflight checklist uses the same operational principle: define observable success and recovery evidence before a change begins.
Practical review checklist
Before approving a retry policy, answer these questions:
- Which exact failures are retryable, and who owns that classification?
- Can repeating the operation create a second business effect?
- What is the maximum attempt count and maximum total elapsed time?
- Is backoff capped, and is jitter appropriate for this caller population?
- Does the policy respect server throttling and the caller's deadline?
- Can retries multiply across layers or overwhelm a struggling dependency?
- What happens after exhaustion, and who is alerted?
- Are retry rates, final outcomes and queue lag visible in telemetry?
Retries are a recovery mechanism, not a substitute for fixing the root cause. A strong policy gives a transient failure a bounded chance to recover while protecting the dependency, preserving business correctness and returning control before the operation's useful deadline expires.