Idempotency Keys: Designing APIs That Survive Retries

Every double-charged customer in history has the same origin story: a client sent a payment request, the network timed out before the response arrived, and someone or something retried. The server processed both. Nobody wrote a bug in the usual sense; everyone behaved reasonably. The system as a whole still took money twice, and now support is issuing a refund while engineering writes a postmortem.

The cure has been industry-standard for a decade: idempotency keys. The client attaches a unique key to each logical operation, and the server guarantees that a given key executes at most once, replaying the original response for any retry. Stripe popularized the pattern, the IETF HTTP API working group drafted a standard header for it, and every payment-adjacent system I have built since uses it. This post covers the full design: HTTP semantics, server storage, concurrency, and the client contract.

Why Retries Are Inevitable, Not Exceptional

The fundamental problem is that a network failure is ambiguous. When your HTTP call times out, you cannot know whether the request never arrived, arrived and failed, or arrived and succeeded while the response got lost. Stripe's engineering blog calls this the classic two generals problem in API form: the client must choose between never retrying, and risking lost operations, or retrying, and risking duplicates.

In practice you do not even get to choose. Mobile networks in Indonesia drop connections mid-flight constantly; load balancers retry on idle timeouts; queue consumers redeliver on at-least-once semantics; impatient users press the button again. Any POST endpoint that moves money, creates orders, or sends messages will be called twice with the same intent eventually. The only question is whether your API is designed to notice.

HTTP Already Solved Half of This

Idempotency is built into most HTTP methods by definition; the gap is POST. That gap is exactly where idempotency keys live:

The Server-Side Mechanics, Step by Step

A robust implementation is a small state machine keyed by the client's idempotency key. The four steps below, and the schema underneath, are the shape I deploy in NestJS on PostgreSQL:

1. On arrival, atomically claim the key: insert a row with status in_progress, using insert-on-conflict-do-nothing so exactly one concurrent request wins the claim.
2. If the insert won, execute the real handler inside the same database transaction as your business writes where possible, then store the response code and body on the key row and mark it done.
3. If the key already exists and is done, compare the stored request hash with the incoming request. Same hash: replay the saved response verbatim. Different hash: reject with 422, because the client is reusing a key for a different operation, which is always a client bug worth surfacing loudly.
4. If the key exists but is still in_progress, return 409 with a retry-after hint. Do not block the connection waiting; the original request may take seconds, and holding workers hostage turns a retry storm into an outage.

-- The storage that makes it work: one row per key attempt
CREATE TABLE idempotency_keys (
  key            text PRIMARY KEY,
  request_hash   text NOT NULL,          -- sha256 of method+path+body
  response_code  int,
  response_body  jsonb,
  status         text NOT NULL DEFAULT 'in_progress',
  created_at     timestamptz NOT NULL DEFAULT now()
);

-- NestJS guard sketch (the real one is an interceptor):
-- 1. INSERT ... ON CONFLICT (key) DO NOTHING RETURNING *
-- 2. inserted?        → run the handler, store response, status='done'
-- 3. conflict + done? → compare request_hash:
--      same   → replay stored response (200, same body)
--      differs → 422: key reused for a different request
-- 4. conflict + in_progress? → 409: original still running, retry later

What Stripe's Implementation Teaches

Stripe's production behavior is a useful calibration target because it has survived more retry traffic than any system you or I will build. Keys are accepted on all POST requests, are not accepted on GET and DELETE since those are already idempotent, and are stored for at least 24 hours, after which a reused key is treated as fresh. Replayed responses return the original result whether it was a success or an error, with an idempotent-replayed header so clients can tell.

Two design choices stand out. First, errors are replayed too: if the original attempt failed validation, the retry gets the same 4xx rather than a second execution. Second, the key is just an opaque string up to 255 characters, with UUIDs recommended; the server attaches no meaning to its contents. Both choices keep the contract simple enough that every client library can implement it identically.

The Four Pitfalls That Reach Production

The Client Contract: One Key per Operation, Not per Request

The server design only works if clients hold up their half: generate one key when the user initiates an operation, then reuse that exact key across every retry of that operation. A new key per HTTP attempt defeats the entire mechanism. In our ERP frontends the key is generated when the confirmation modal opens, so the approve button can be hammered, the network can flap, and the backend still books exactly one journal entry:

// Client side: generate ONE key per logical operation,
// reuse it across every retry of that operation.
const idempotencyKey = crypto.randomUUID()

async function createPayment(payload: PaymentDto) {
  return retryWithBackoff(() =>
    fetch("/api/payments", {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        "Idempotency-Key": idempotencyKey, // same key on every attempt
      },
      body: JSON.stringify(payload),
    })
  )
}

Implementation Checklist

• Accept an Idempotency-Key header on every unsafe POST; reject requests without it on critical routes.
• Claim keys atomically with a single insert-on-conflict statement; never check-then-insert.
• Store request hash, response code, and response body; replay errors as faithfully as successes.
• Return 409 for concurrent in-progress retries and 422 for key reuse with a different payload.
• Expire keys on a schedule, document the window, and emit a replayed marker header so clients can observe deduplication happening.

Design for the Retry, Because It Is Coming

Idempotency keys are one of those patterns with an enormous ratio of value to code: a table, an interceptor, and a header turn the scariest class of distributed-systems bug into a non-event. The takeaway is to treat retries as a certainty of operating on real networks, especially on the mobile connections most Indonesian users live on, and to make at-most-once execution a property your API guarantees rather than a behavior you hope for. Build it once, wire it into every POST that matters, and the double-charge postmortem becomes someone else's story.