How Kafka Log Compaction Keeps Only the Latest Value per Key (and Where It Bites You)

The forwarding-address register

In the Kafka and messaging reliability post, Ravi ran a sorting office where every letter was an event. This post looks at one drawer in that office: the forwarding-address register. (Ravi, his office and every number below are made up for teaching.)

Every time a customer moves house, a change slip goes into the drawer: "Customer 42 now lives at 7 Lake Road." After a few years the drawer holds millions of slips, but the clerks only care about one thing per customer: the newest address. Asha's three old addresses are just taking up space.

So once a week a clerk goes through the drawer, keeps the newest slip for each customer, and throws the older ones away. If a customer closes their account, a special slip goes in that says "Customer 42: no address, forget this person." After a while, that slip goes too.

That clerk is Kafka's log cleaner, the drawer is a compacted topic, and the "forget this person" slip is a tombstone. This post explains what the cleaner really does, when it runs, and the edge cases that interviewers like to ask about.

In 30 seconds, as an architect would frame it: "A compacted topic is a changelog, not a queue. Kafka keeps at least the latest record for each key and deletes older records with the same key in the background. Order within a partition and offsets are preserved, so a new consumer can replay from offset zero and rebuild current state. It is not a key-value store: there is no lookup by key, duplicates exist until the cleaner runs, and a delete is only final after the tombstone itself expires. I use it for state that must be rebuilt: Kafka Streams changelogs, CDC snapshots and service configuration."

The idea in simple words

A normal Kafka topic uses time- or size-based retention (cleanup.policy=delete): after seven days, say, whole old files are deleted, no matter what is in them. That is right for events like "order placed", where every event matters and old ones stop mattering.

A compacted topic (cleanup.policy=compact) uses a different rule: delete a record only when a newer record with the same key exists later in the same partition. The result is a log that never stops being a complete picture of "the current value of every key", however old the topic gets.