The message arrives twice — the design decides whether that matters
3 min read
The difference between At-least-once and Exactly-once is not whether a duplicate arrives — in both designs it does — but whether anything stops that duplicate from producing a second effect. At-least-once delivery redelivers a message whenever an acknowledgement goes missing, and the handler runs its side effect again; "exactly-once" is that same redelivery with a deduplication gate placed in front of the side effect, so the message still lands twice but the charge, the row write, or the email happens once. On the wire the two designs are identical. The only thing that changes is what absorbs the copy.
Why the same message arrives twice
A broker cannot tell a lost message from a lost acknowledgement. The handler runs balance = balance - 20, commits, and sends its ack — and then the ack dies: the process crashes after the commit, a network partition swallows the reply, or a consumer-group rebalance times the session out. From the broker's side these all look the same as a handler that never ran, so it does the only safe thing and redelivers. The redelivered message reaches the handler, the effect fires a second time, and the customer is charged twice.
The ordering is the whole trap. The effect commits before the ack, so the gap between the two is a window in which a redelivery produces a second, real effect. This is at-least-once delivery, and it is the honest default of every durable broker: it would rather hand you the message twice than lose it once.
Producer
publishes charge order-88; the broker persists it
Handler
charges the card, commits — then the ack is lost to a crash or rebalance
Broker
the delivery is still outstanding, so on timeout it redelivers the same message
Handler
at-least-once: no memory of order-88, so it charges the card a second time
Handler
exactly-once: the gate has already seen order-88 and skips the effect — charged once
The redelivery happens in both designs. Only the dedupe gate on the idempotency key stops the second charge.
Why exactly-once delivery is impossible
Push on the word and it comes apart. True exactly-once delivery is impossible over an unreliable network — it is the Two Generals Problem: no finite exchange of messages can leave both sides certain the last one arrived, so the sender can never safely stop retrying, and retrying is what creates duplicates. What ships under the "exactly-once" label is therefore not a stronger transport guarantee. It is at-least-once delivery plus message deduplication: the duplicate still crosses the wire, and a gate keyed on an idempotency key — supplied by the producer, the same order-88 on every redelivery — recognises the key it has already processed and drops the second copy before it does any work.
The honest name is "effectively once", and the distinction is load-bearing. Kafka's exactly-once semantics is exactly this shape with the machinery moved inside the broker: an idempotent producer is assigned a producer ID, the broker deduplicates on a per-partition sequence number, and transactions make the consume-process-produce triple atomic. Its scope is Kafka-to-Kafka — it does not reach out and make an external database write or a payment API call happen once. That part is still yours to guard.
When at-least-once is enough
Reach for plain at-least-once when the side effect is already idempotent or harmless to repeat: setting a value rather than incrementing it, overwriting a cache entry, upserting a row by primary key, or sending a notification the user can tolerate seeing twice. If replaying the message leaves the system in the same state, the duplicate costs nothing and a dedupe store is pure overhead you have to operate.
Add the idempotency key when the effect is not naturally repeatable and the second firing is expensive or irreversible — charging a card, shipping an order, transferring funds, incrementing a counter. Then the gate earns its keep, with one parameter to watch: the dedupe store needs a TTL long enough to outlive the broker's redelivery window, or a late redelivery slips past an expired key and double-charges anyway.
At-Least-Once vs Exactly-Once in a system design interview
The probe is usually "how do you make sure this consumer processes each message once?" The trap answer is "I'll turn on exactly-once." The crisp answer names the mechanism: you cannot get exactly-once delivery, so you accept at-least-once and make the effect idempotent — attach a producer-supplied idempotency key to each message, check it against a dedupe store before doing the work, and skip the effect when the key is already present. Be ready for the follow-ups: the key comes from the producer, not the consumer, so a retry carries the same one; the dedupe store's TTL is a correctness parameter, not a cleanup convenience; and a broker's own exactly-once feature covers its own topics, never your downstream database or third-party API. Say that last part and you have shown you know where the guarantee stops.
