Idempotency for write APIs: surviving retries without duplicate harm
Treat duplicate delivery as normal: idempotency keys for money paths, resource-oriented verbs where they fit, and idempotent consumers when messages are at-least-once.
Idempotency means you can invoke the same write more than once and the system still ends up in a sane state—no double charge, no duplicate inventory reservation, no second “welcome email” that makes HR think you joined twice.
Networks retry. Users double-click. Gateways replay. If your write path assumes “this HTTP request only happens once,” you are one timeout away from fiction—Stripe’s idempotency keys are famous for a reason; your internal POST /orders deserves the same seriousness.
Idempotency for writes
1. Use an idempotency key from the client
For operations where intent matters more than the exact packet shape, have the client generate a stable unique key per logical action (UUID is fine) and send it on every attempt—classically Idempotency-Key on POST-like writes.
Rules of thumb:
- One key maps to one business outcome (“pay invoice 42,” not “every HTTP try forever”).
- The client should reuse the same key on retry, not mint a fresh one per attempt.
- If the client cannot generate keys (some legacy browsers, embedded devices), you need another dedupe strategy—but most backends can do better than “hope.”
2. Store idempotency keys with the result
The server needs a durable record keyed by that token: status, optional request fingerprint, response body (or pointer), and timestamps.
A boring flow:
- If key unknown, insert a row as
PROCESSING(or acquire a lock), do the side effect, persist response, markCOMPLETED. - If key known and
COMPLETED, return the stored response without redoing work. - If key known and
FAILED, decide whether retries are allowed and whether you return the same error or a new one—document the behavior.
Use a transaction or unique constraint so two app servers cannot both “win” the first insert for the same key.
3. Handle concurrent requests safely
Two identical requests can arrive in the same millisecond. If your handler is “read then insert,” both threads can decide “no row yet” and charge twice.
Mitigations that actually work in production:
- Unique index on idempotency key so only one insert succeeds; the loser reads the winner’s row.
- Short advisory lock or lease row keyed by hash if you cannot tolerate insert races.
- Return 409 or a structured “already processing” if you want the client to back off—just be consistent.
4. Deal with in-progress work and crashes
If the first request dies after charging money but before you persist COMPLETED, a retry must not charge again. That is why “process then update” ordering matters, and why payment processors obsess about reconciliation and outbox patterns.
Practical statuses:
PROCESSING— you accepted responsibility; retries should wait or get a “still working” response.COMPLETED— return cached success payload.FAILED— return a stable error; only allow retry if the operation is genuinely safe to attempt again.
Document what happens if a client gives up and submits a new key for the same intent—that is a product question, not just an engineering detail.
5. Set a sensible lifetime for idempotency records
You cannot keep every key forever without turning the dedupe table into its own database product.
Pick a TTL window that matches how long retries happen in the wild—often 24–72 hours for HTTP, sometimes longer for batch reconcilers. Archive or delete after expiry, with metrics so you notice if clients still hit old keys.
Once keys expire, anything still retrying is basically a new intent—same story as when a Kafka consumer replays from an offset you did not expect: the handler had better be safe under duplicates.
Push heavy work async
Reality check: “exactly once”
Exactly-once delivery across distributed components is not something you buy off the shelf—you get at-least-once with idempotent handlers, or you accept data loss. Design for duplicates everywhere messages exist (queues, webhooks, outbox flushes) and make handlers safe to replay.
Summary table
| Approach | How it works | Best for | Watch out for |
|---|---|---|---|
| Idempotency-Key | client token + server store + replay | payments, orders, bookings | storage, races, TTL, in-flight states |
| Natural idempotency | PUT/PATCH on stable IDs, compare-and-set | status changes, config | not all domains map to CRUD |
| Idempotent consumers | business key / dedupe table in worker | queues, webhooks, CDC | poison messages, ordering, backfill |
In short
- Assume duplicates on every write path that touches money, inventory, or compliance.
- Return the same response for the same idempotency key once work is done.
- Use constraints so concurrency cannot create two “first” outcomes.
- Expire keys on a schedule that matches real retry behavior.
- Async == at-least-once in practice—make consumers boring under replay.
If you can answer “what happens if this runs twice?” for each write, you have idempotency. If you cannot, you have hope—and hope is not a storage engine.