Redis and Kafka are often discussed as if they were competing ways to make an application faster. In a real-time system, the more useful question is what each component is responsible for—and what must still be true when a process crashes, a consumer falls behind or a client reconnects.
A low-latency read and a durable event history solve different problems. Redis is commonly used for in-memory data access, caching and selected coordination patterns. Apache Kafka is commonly used to retain and distribute ordered streams of events so independent consumers can process them at their own pace. Both products have capabilities beyond those shorthand descriptions, but they should not be treated as interchangeable by default.
This guide uses an auction and notification workflow to make the trade-offs concrete. The aim is not to prescribe a technology stack for every system; it is to choose based on data semantics, recovery requirements and observed bottlenecks.
Start with the responsibility
Before comparing features, write down what the component must guarantee.
| Question | Redis is often a good fit when… | Kafka is often a good fit when… |
|---|---|---|
| Primary access pattern | The application needs fast key-based reads, updates, expiry or selected atomic operations. | Producers append events and independent consumers process a stream. |
| Data lifetime | Data is cached, derived, short-lived or stored under an explicitly chosen Redis persistence model. | Events must remain available for a configured retention period and be read again by consumers. |
| Consumer model | A request needs a fast lookup, or a suitable Redis data structure coordinates a small operation. | Multiple consumer groups need independent progress through the same event history. |
| Recovery question | Can the value be rebuilt, replicated or restored under the selected Redis configuration? | Can a consumer resume from offsets and replay retained events safely? |
| Ordering boundary | Operations can be handled by the chosen key, data structure and atomicity model. | Ordering is needed for records with the same key and partition. |
| Main operational concern | Memory sizing, eviction, persistence, replication, failover and hot keys. | Partition design, replication, retention, consumer lag, rebalances and replay. |
These are common patterns, not product limits. Redis Streams can support append-oriented stream processing, and Kafka can support low-latency event delivery. The correct comparison is between the specific data structure, delivery semantics and operational configuration you intend to use.
Worked example: an auction accepts a bid
Imagine a bidder submits a bid of ₹10,500. The system must decide whether it is valid against the authoritative auction state. After that decision commits, several other actions may follow: update a live view, notify participants, refresh reporting and record an audit event.
A defensible flow separates the decision from the downstream reactions:
- The bid service validates the bidder, auction status and amount.
- A database transaction applies the authoritative state transition and records the event intent.
- An outbox publisher forwards the committed event to Kafka when durable event delivery is required.
- Consumers update projections, send notifications or feed analytics independently.
- Redis may accelerate reads of current auction state or hold a rebuildable projection when its consistency policy is explicit.
- The real-time layer sends updates after the authoritative decision; reconnecting clients can fetch a fresh snapshot.
The database transaction and concurrency strategy decide whether the bid was accepted. A cache entry, a Kafka message or a WebSocket notification must not independently decide the winner.
For the concurrency boundary and closing-time race, see Handling Concurrent Bids in a Real-Time Auction System and the Auction Contention Estimator.
Where Redis earns its place
1. Fast reads and rebuildable projections
Redis can reduce repeated database work for hot values such as a current auction summary, feature flags, rate-limit counters or session-related data. The design still needs a freshness policy: a cache hit is useful only if the application understands how stale the value may be and how it is invalidated or refreshed.
A cache should have a clear miss path. If Redis is unavailable or an entry expires, can the application safely read from a durable source? If the answer is no, the value may be business-critical state rather than a disposable cache, and the durability and recovery design deserves a separate review.
2. Atomic operations and coordination
Redis offers atomic commands and data structures that can help with counters, rate limiting, short-lived coordination and selected locking patterns. Atomicity of one command does not automatically make a multi-step business workflow safe. Lock ownership, expiry, fencing, retries and failure recovery still need deliberate treatment.
Do not move a database consistency problem into Redis simply because a command is fast. Define the invariant first, then choose a mechanism whose guarantees match it.
3. Pub/Sub and Redis Streams are not the same
Redis Pub/Sub is useful for transient notifications where disconnected subscribers do not need a retained history. It does not provide a durable replay log for subscribers that were offline.
Redis Streams add retained entries and consumer-group features, so they can be appropriate for some event-processing workloads. Their operational and delivery model still needs to be evaluated against the requirements for retention, independent consumers, replay, ordering, scale and recovery. The fact that a technology supports streams does not remove the need to specify those guarantees.
Where Kafka earns its place
1. Independent consumers and event history
Kafka is useful when one business fact needs to drive multiple independently operated workflows. A committed bid.accepted event can feed notifications, audit processing, analytics and search projections without forcing the bid API to wait for each consumer.
Consumer groups allow each logical consumer application to track its own progress. Retained events can be replayed when a projection needs rebuilding or a consumer bug has been fixed, provided the necessary records are still within retention and the replay is safe.
2. Ordering is scoped, not global
Kafka guarantees ordering within a partition. If events for one auction must be processed in order, use a stable key such as auctionId and design the partitioning and consumer logic accordingly. Events with different keys can be processed in parallel, but a single hot key can still become a bottleneck.
Increasing the number of partitions does not split one key's ordered stream across many consumers. It can increase parallelism across keys, but a hot partition or slow downstream dependency may remain the limiting factor.
3. Consumer lag is an operational signal
Lag means consumers have not yet processed all available records. A non-zero value is not automatically an incident; the useful questions are whether lag is growing, how quickly it can drain, whether the oldest unprocessed records are approaching retention limits, and whether downstream work is still meeting its service objectives.
Measure both the incoming record rate and sustained processing rate. If consumers process 8,800 records per second while producers add 6,000, the idealized net drain rate is 2,800 records per second. A backlog of 2.4 million records would then take about 14.3 minutes to drain, assuming those rates hold and there is no partition skew or rebalance interruption.
Use the Kafka Consumer Lag Drain-Time Calculator to model this relationship. It is a planning estimate, not a replacement for broker metrics, per-partition lag, consumer telemetry or load tests.
A practical decision sequence
Choose Redis first when the problem is a fast data access pattern
Examples include a rebuildable cache, a rate-limit counter or a low-latency lookup. Define what happens on a cache miss, how data is invalidated, whether eviction is acceptable and how the system behaves during failover.
Choose Kafka when the problem is durable asynchronous distribution
Examples include publishing a committed order or bid event to several independent consumers, replaying retained events to rebuild a projection, or decoupling notification processing from a transaction API. Define event contracts, keys, retention, retries, idempotency and ownership of failed messages.
Use both when the responsibilities are genuinely different
A system can use Redis to serve a current-state projection quickly and Kafka to distribute the committed events from which downstream projections are updated. This is not automatically better: it adds a second state representation, and therefore a freshness, reconciliation and recovery problem. Use it when the latency or workload benefit justifies the operational cost.
For broader workload sizing, use the System Design Capacity Calculator and the network infrastructure capacity planning guide. For availability objectives and error-budget policy, see the SLO error-budget planning guide and SLO Error Budget Calculator.
Failure scenarios to design before launch
| Scenario | Question the design must answer |
|---|---|
| Redis restarts or evicts an entry | Can the value be rebuilt, and is stale data clearly handled? |
| Kafka consumer crashes after an external side effect | Can redelivery cause duplicate email, payment or other business action? |
| Database commit succeeds but event publication fails | Is there an outbox or another reliable reconciliation path? |
| Consumer lag keeps growing | Is processing capacity below incoming load, or is a dependency, rebalance or hot partition slowing the group? |
| A consumer must replay old events | Are the records retained, and are handlers safe to replay? |
| A cache and database disagree | Which source is authoritative, and how is the projection repaired? |
| One key receives disproportionate traffic | Can the workload be partitioned safely, or does the invariant require serialization? |
Kafka producer idempotence helps with a specific part of publishing; it does not guarantee that every external side effect happens exactly once. Consumer handlers should use idempotency keys or durable deduplication where required. Likewise, a Redis lock is not a substitute for a database constraint or transaction when the business invariant is enforced by the database.
Operations and security still matter
The technology choice does not replace operational discipline. Define service ownership, capacity thresholds, alerts, recovery steps and change controls for each component.
- For Kafka, observe per-partition lag, incoming and processing rates, rebalance frequency, failed records, retry queues and the age of the oldest unprocessed work.
- For Redis, observe memory pressure, evictions, latency, hot keys, persistence status, replication health and failover behaviour.
- For both, test dependency loss, bounded retries, duplicate handling, restore or replay procedures and the effect of partial failure on user-visible operations.
The platform also needs secure access controls, protected credentials, appropriate network boundaries and careful handling of sensitive event fields. The Cloudflare and Nginx origin-security guide covers the edge-to-origin boundary; the AKS production-readiness guide provides a broader way to assess workload, dependency and recovery risk before platform changes.
A short architecture checklist
Before approving a Redis/Kafka design, make sure the team can answer:
- What is the authoritative source of business state?
- Which values are caches or derived projections, and how are they rebuilt?
- Does the workflow need retained events, independent consumer progress or replay?
- What ordering is required, and at what key or partition boundary?
- How are duplicate requests, messages and external side effects handled?
- What is the expected incoming rate, processing capacity and acceptable lag?
- How will the system behave when a dependency is unavailable?
- Which metrics, alerts and recovery procedures are owned by the operating team?
The right answer may be Redis, Kafka, Redis Streams, a simpler queue, a database-backed worker—or a combination. Start with the guarantee the business needs, then choose the smallest architecture that delivers it and can be operated confidently.
Related engineering notes
- Designing an Event-Driven Notification Architecture with Apache Kafka — event contracts, retries, acknowledgements and downstream delivery.
- Kafka Consumer Lag Drain-Time Calculator — estimate whether consumers can drain a backlog while new records continue to arrive.
- Handling Concurrent Bids in a Real-Time Auction System — protect authoritative state when requests compete.
- SLO error-budget planning — connect operational signals to service objectives and release decisions.