Designing an Event-Driven Notification Architecture with Apache Kafka

Modern platforms generate notifications from almost every critical business workflow.

A user signs in and receives an OTP. A bidder submits an EMD payment. An auction reaches an important stage. A wallet withdrawal succeeds or fails. A refund is processed.

The simplest implementation is often to send an email or SMS directly from the service performing the business operation. That works initially.

At scale, however, notification delivery becomes a distributed-systems concern of its own. Business services become coupled to external providers. Provider latency starts affecting API response time. Retry logic gets duplicated across services. Failures become harder to trace, and adding a new communication channel requires changes across multiple domains.

A better boundary is to treat notification as a reaction to a business event rather than part of the business transaction itself.

This article describes an event-driven notification architecture using Apache Kafka, based on patterns I have used while designing microservice platforms with user, auction, payment, wallet and notification domains.

The objective is not simply to introduce Kafka. It is to examine the engineering decisions required to make asynchronous notification delivery reliable, observable and operationally manageable.

The architectural problem

Consider an EMD payment workflow in an auction platform.

Once the payment service confirms the payment, several actions may follow:

• update the payment state,
• make the new state available to the auction workflow,
• notify the bidder,
• send an email,
• send an SMS,
• record notification history,
• potentially notify other interested systems.

A tightly coupled implementation might look like this:

This creates several problems.

If the SMS provider is slow, should the payment request become slow?

If email delivery fails, should a successfully completed payment be reported as failed?

Should the Payment Service understand email templates, SMS gateways and provider-specific response codes?

And when another notification channel is introduced, should every business service be changed?

The answer to these questions points toward a separate asynchronous boundary.

Separate business completion from notification delivery

The business service should own the business transaction. The notification service should own notification delivery.

Once the business operation reaches a valid state, the producer publishes a domain event:

The payment operation no longer waits for an external email or SMS provider. More importantly, the Payment Service does not need to know how the notification is delivered. That responsibility belongs to the Notification Service.

Domain events, not provider commands

One of the most important design decisions is the meaning of an event.

Avoid designing Kafka messages as thin wrappers around provider commands such as:

send-email
send-sms

That moves implementation details into the event contract.

A more useful event expresses what happened in the business domain:

emd.payment.confirmed
wallet.withdrawal.succeeded
wallet.withdrawal.failed
auction.started
user.login.requested

The Notification Service can then determine which channels and templates apply.

In an existing platform, topic naming may already follow conventions such as:

user.login.notify
emd.payment.notify
auction.intimation.letter.notify
wallet.withdraw.success.notify
wallet.withdraw.fail.notify
wallet.remit.success.notify
wallet.remit.fail.notify
wallet.refund.success.notify
wallet.recon.exception.notify
wallet.daily.status.notify

The exact naming convention is less important than having a consistent ownership model. The event should communicate a business fact or notification intent without exposing SMTP, SMS-provider or push-provider implementation details.

Define an explicit event contract

Asynchronous systems need contracts just as much as REST APIs do.

A notification event should carry enough context for the consumer to process it without repeatedly calling the producer for basic information.

A simplified contract could look like this:

{
  "eventId": "evt-9f8d41",
  "eventType": "EMD_PAYMENT_CONFIRMED",
  "eventVersion": "1.0",
  "occurredAt": "2026-10-01T10:30:00Z",
  "source": "payment-service",
  "correlationId": "payment-78431",
  "recipient": {
    "userId": "USR-1028",
    "email": "[email protected]",
    "mobile": "+91XXXXXXXXXX"
  },
  "channels": ["EMAIL", "SMS"],
  "template": "EMD_PAYMENT_SUCCESS",
  "data": {
    "auctionId": "AUC-451",
    "amount": 25000,
    "currency": "INR"
  }
}

Several fields deserve particular attention.

eventId

Every event should have a unique identifier. It becomes important when handling retries and duplicate delivery.

eventType

This identifies the business meaning of the event. Consumers should not have to infer intent from arbitrary payload fields.

eventVersion

Event contracts evolve. Versioning provides a controlled way to introduce compatible changes and eventually retire older contracts.

correlationId

A distributed transaction may cross multiple services. A correlation identifier allows logs, Kafka events and notification history to be connected during troubleshooting.

data

Keep template-specific variables separate from transport metadata. This makes the event easier to validate and the rendering process easier to evolve.

Topic strategy

There is no universally correct rule for Kafka topic granularity.

Creating one topic for every tiny event can create operational overhead, while placing unrelated events into a single topic can make ownership and scaling difficult.

For a notification architecture, useful boundaries normally follow business ownership and operational characteristics.

For example:

user.login.notify
emd.payment.notify
auction.*.notify
wallet.*.notify

A topic strategy should consider:

• event ownership,
• expected volume,
• retention requirements,
• ordering requirements,
• consumer scaling,
• access control,
• failure isolation.

Topic naming should therefore be a deliberate architecture decision rather than an incidental implementation detail.

Partitioning and ordering

Kafka ordering is guaranteed within a partition, not globally across an entire topic. That makes the message key important.

For example, wallet events may be keyed using walletId, while auction events may use auctionId.

Events sharing the same key are routed consistently to the same partition, allowing their relative order to be preserved.

The key should represent the business entity for which ordering matters. Global ordering is rarely worth the scalability cost.

Consumer groups provide horizontal scalability

The Notification Service can run multiple consumer instances as part of the same consumer group. Kafka distributes topic partitions among those consumers.

This allows notification processing capacity to grow independently of the services generating the events.

If auction traffic increases dramatically during a high-volume event, additional notification consumers can be introduced without scaling the Payment Service or User Service.

That independent scaling is one of the major advantages of the asynchronous boundary.

Notification orchestration

After consuming an event, the Notification Service performs several responsibilities.

Provider-specific logic should remain behind adapters.

The domain processing code should not depend directly on a particular vendor's API. That keeps provider replacement and multi-provider strategies manageable.

Delivery history is part of the architecture

Sending a notification is only half of the problem. Operational systems also need to know what happened afterwards.

A useful notification history record can capture:

eventId
eventType
recipient
channel
template
provider
attemptCount
status
providerMessageId
providerResponse
createdAt
sentAt
failedAt

Possible states might include:

QUEUED
PROCESSING
SENT
FAILED
RETRY_PENDING

For channels or providers that support acknowledgement callbacks, the provider result can update the final delivery state.

This becomes especially important in regulated or transaction-oriented platforms where operators may need to answer: Was the notification generated? Which channel was attempted? When was it sent? Did the provider accept it? How many attempts were made?

The answers should come from structured delivery history rather than application log archaeology.

Failure handling must be designed explicitly

Distributed systems fail. Email providers become unavailable. SMS gateways time out. DNS resolution fails. Connections reset. Payloads can be invalid.

The architecture therefore needs to distinguish transient failures from permanent failures.

A temporary provider timeout may justify a retry. An invalid recipient or malformed event may not.

Retries should always be bounded. An infinite retry loop can turn one bad event into a permanently blocked consumer pipeline.

Retry topics and dead-letter handling

For systems requiring stronger recovery isolation, retry topics can separate delayed retry traffic from the main event stream.

This is an architectural option rather than a requirement for every deployment.

A dead-letter topic is useful only when there is an operational process behind it. Messages reaching the DLQ should be observable and actionable.

Teams need to know why the event failed, how many attempts were made, whether it can be safely replayed, whether data must be corrected first, and who owns resolution.

A DLQ that nobody monitors simply hides failures in another Kafka topic.

Assume that a consumer may see an event again

A critical principle in event-driven systems is:

Design consumers so that processing the same event again does not create an incorrect business outcome.

Kafka commonly operates with at-least-once delivery semantics, so duplicate processing must be considered.

For notification delivery, duplicate processing can be particularly visible: the user may receive the same OTP, payment confirmation or refund message multiple times.

A simple idempotency strategy can use the event identifier together with the delivery channel:

eventId + channel

Before sending:

In a database, a unique constraint around the idempotency key provides stronger protection than an in-memory check.

Idempotency should therefore be treated as a business-processing concern, not only as a Kafka producer setting.

Producer idempotence and consumer idempotence are different

Kafka supports an idempotent producer. That capability protects against duplicate records being written to Kafka because of producer retries.

It does not automatically guarantee that an external side effect performed by a consumer occurs exactly once.

After recovery, Kafka may deliver the event again. The email provider does not know that the first delivery was already successful unless the application maintains an idempotency mechanism.

This distinction is important when designing systems that interact with databases, payment gateways, email providers or other external systems.

When should the consumer commit the offset?

Offset management determines when Kafka considers an event processed.

For notification processing, committing before the external operation creates a risk:

Kafka believes the event was processed even though the notification failed.

Committing after successful processing provides stronger reliability:

But this introduces the possibility of redelivery if the process crashes after the side effect and before the commit.

That is why offset strategy and idempotency must be designed together.

Acknowledgements should be asynchronous too

Some providers return only an acceptance response when a message is submitted. Final delivery status may arrive later through another mechanism.

The notification architecture should distinguish accepted by provider from delivered to recipient.

In systems where acknowledgements themselves are processed through messaging, dedicated acknowledgement events can provide the same decoupling and auditability as the original notification pipeline.

Observability across the complete event path

Asynchronous architecture removes runtime coupling, but it also removes the simplicity of following a single HTTP request through one process.

Observability therefore becomes more important.

At minimum, I want to be able to trace:

Useful metrics include:

events produced
consumer lag
notifications processed
notifications succeeded
notifications failed
retry count
DLQ count
provider latency
channel-specific failure rate

Consumer lag deserves particular attention.

A healthy API does not necessarily mean a healthy notification system. Kafka may be accepting events while consumers fall progressively behind.

Alerting on lag and failure rates provides earlier visibility into this type of degradation.

What happens when the Notification Service is unavailable?

This is where the asynchronous design becomes valuable.

Business services can continue publishing events as long as Kafka itself remains available.

When notification consumers recover, they resume processing from their offsets.

The outage therefore affects notification latency rather than forcing every upstream business transaction to fail.

That is failure isolation.

It is one of the strongest reasons for introducing an event broker between transactional services and secondary workflows.

Kafka does not remove every consistency problem

Introducing Kafka does not automatically make a system reliable.

One important failure window remains:

The business state has changed, but the event was never published.

For workflows where losing that event is unacceptable, an architecture can introduce the Transactional Outbox Pattern.

The database change and outbox record succeed or fail together. A separate publisher then reliably forwards pending outbox events to Kafka.

Not every notification warrants this complexity, but critical payment, wallet or auction events may justify it.

Security boundaries still matter

Kafka should not become an unrestricted internal event bus.

Production designs should consider:

• authenticated producers and consumers,
• encrypted broker communication,
• topic-level authorization,
• protection of recipient information,
• secret management,
• retention policies,
• auditability.

Notification payloads often contain email addresses, mobile numbers or transaction context.

Only the information required by the consumer should be placed in an event. Avoid turning Kafka into a replicated store of unnecessary sensitive data.

Schema governance becomes important as the platform grows

JSON payloads are convenient when a system is small.

As producer and consumer counts increase, schema evolution becomes an architecture concern.

Options such as Avro, Protobuf or JSON Schema with a schema registry can provide stronger contract governance.

The important principle is not the particular serialization technology. It is that producers should not be able to make uncontrolled contract changes that silently break consumers.

A mature event platform defines ownership, compatibility rules, versioning, validation and deprecation policy before event contracts become difficult to manage.

Architecture decisions and trade-offs

Event-driven notification architecture introduces real advantages:

• business services are isolated from notification providers,
• notification processing scales independently,
• temporary consumer outages do not immediately break upstream transactions,
• new channels can be introduced behind the notification boundary,
• failures and retries can be handled centrally,
• delivery history provides operational traceability.

But Kafka also introduces cost:

• more infrastructure,
• asynchronous debugging,
• event contract governance,
• offset management,
• duplicate handling,
• operational monitoring,
• replay complexity.

Kafka should therefore be introduced because the system benefits from asynchronous decoupling, durability and independent scaling—not simply because microservices are present.

For a small application sending a handful of emails, a queue or background worker may be entirely sufficient.

For a transaction-heavy platform with user, auction, payment and wallet workflows generating notifications independently, the event-driven boundary becomes considerably more valuable.

Final architecture

Around that flow sit the operational controls:

Event contracts
Topic ownership
Partition strategy
Consumer groups
Retry policy
Idempotency
Failure isolation
Observability
Security
Schema governance
Replay / recovery

Those controls are what turn a Kafka diagram into a production architecture.

Closing perspective

The most important decision in this architecture is not Kafka itself.

It is the boundary.

A Payment Service should know that an EMD payment succeeded.

It should not need to know how an email provider authenticates, how an SMS template is rendered, whether a push provider is temporarily unavailable, or how many times a notification has been retried.

Publishing a well-defined event allows each service to remain focused on its own responsibility.

Kafka provides the durable asynchronous bridge.

The Notification Service owns communication.

And explicit contracts, idempotency, retries, acknowledgements and observability make that bridge dependable enough for real distributed systems.

Related real-time transaction design

For auction platforms, the event boundary comes after the bid decision is committed. Read Handling Concurrent Bids in a Real-Time Auction System for the authoritative transaction, idempotency and outbox considerations that make downstream events trustworthy.

Consumer throughput and recovery

A Kafka consumer group should be sized against both incoming records and the backlog it needs to drain—not just the number of consumers. Use the Kafka Consumer Lag Drain-Time Calculator to estimate net drain rate, partition-limited parallelism and the consumer count implied by a recovery target. The model is deliberately simple; validate it against per-partition lag, rebalances and sustained processing telemetry.

For the broader choice between fast state access and a retained event stream, read Redis vs Kafka: Choosing the Right Role in a Real-Time Architecture.