Kafka and Messaging Reliability: One Diwali Night in Ravi's Sorting Office

8 pm, the night before the Diwali sale

Ravi runs the sorting office for a shopping app. (The app, Ravi and every number in this story are made up for teaching.) Every time a customer places, pays for or cancels an order, a letter arrives at Ravi's office. Clerks read the letters and do the work: they pack the parcel, send the SMS, update the stock count.

Tomorrow the sale starts. Ravi expects ten times the usual letters. Tonight he has to make sure of three things:

1. No letter is lost, even if a building loses power.
2. No letter is acted on twice, because nobody should be charged twice.
3. One bad letter never stops the whole office.

Those are the same three promises a messaging system like Apache Kafka has to keep. We will follow Ravi through the night. Each problem he hits is a real Kafka failure, and each fix is a real Kafka setting or pattern.

In 30 seconds, as an architect would frame it: "Kafka is a replicated, append-only log. I key messages by the entity whose order matters, so they share a partition. I make writes durable with replication factor 3, acks=all and min.insync.replicas=2. I assume at-least-once delivery and make consumers idempotent, because end-to-end exactly-once stops at Kafka's edge. Transient failures get bounded retries with backoff and jitter, and poison messages go to a watched dead-letter topic with a replay tool. Then I watch the lag."

This guide builds on Architecture, messaging and patterns.

1. The office layout: pigeonholes, slot numbers and bookmarks

Ravi's office has a wall of numbered pigeonholes. A letter never goes into "any free slot". The office looks at the order number on the envelope and always files that order's letters into the same pigeonhole. Inside a pigeonhole, letters stack strictly in arrival order, and each gets a slot number: 0, 1, 2, 3 and so on.

Clerks work in a team. Each pigeonhole belongs to exactly one clerk in the team at a time. Each clerk keeps a bookmark for each pigeonhole: "I have handled everything up to slot 41." Letters are not thrown away after reading. Ravi keeps them for seven days, so a new team (say, the analytics team) can start from slot 0 and read the whole week.

Here is the same office in Kafka's words: