Designing Reliable Webhooks: Retries, Signing, and Replay

Webhooks look simple from the outside: your system detects an event, sends an HTTP POST to a URL the consumer registered, and moves on. In practice, that one sentence hides a distributed systems problem. The consumer's server might be down for maintenance. The network might drop the response after the consumer already processed the event. A deploy might restart your delivery worker mid-request. Every one of these failure modes is routine, not exceptional, and a webhook system that ignores them will eventually deliver an event twice, lose an event forever, or let an attacker forge a payload that looks legitimate.

This post walks through the four pillars of a webhook system that consumers can build production integrations on top of: signing every payload so consumers can verify authenticity, retrying failed deliveries with backoff instead of hammering a struggling endpoint, giving consumers a way to safely handle duplicate deliveries, and keeping a durable event log so both sides can replay history when something goes wrong. None of this is exotic. It is the same handful of patterns that payment processors and API platforms have converged on independently.

Why at-least-once delivery is the only honest guarantee

There are three delivery guarantees you could theoretically offer: at-most-once, exactly-once, and at-least-once. At-most-once means you might silently drop events, which is unacceptable for anything a consumer relies on for billing, inventory, or state sync. Exactly-once sounds ideal but is not achievable across an unreliable network boundary without the consumer's cooperation, because you cannot know whether your request succeeded if the acknowledgment gets lost in transit. That leaves at-least-once as the only guarantee you can honestly make: every event is delivered one or more times, and it is the consumer's job to treat duplicate deliveries as a normal occurrence rather than a bug.

• Design the sender assuming every delivery attempt might be repeated, not assuming it will be sent exactly once.
• Document the guarantee explicitly in your webhook docs so consumers build deduplication from day one instead of discovering it in production.
• Give every event a stable, unique identifier that stays the same across retries, so consumers have something concrete to deduplicate against.

Signing payloads so consumers can trust the sender

Anyone can send an HTTP POST to a public URL, so a consumer's endpoint needs a way to confirm a request actually came from you and was not forged or replayed by an attacker who intercepted an old delivery. The standard approach is HMAC: you and the consumer share a secret, and every outgoing request includes a signature computed by hashing the request body together with that secret and a timestamp. The consumer recomputes the same hash on their end and compares it to the signature you sent. If they match, the payload is authentic and untampered; if they do not, the consumer rejects the request outright.

// Signing an outgoing webhook payload — Node.js
import crypto from "node:crypto"

function signPayload(rawBody: string, secret: string, timestamp: number) {
  const signedPayload = `${timestamp}.${rawBody}`
  const signature = crypto
    .createHmac("sha256", secret)
    .update(signedPayload)
    .digest("hex")
  return `t=${timestamp},v1=${signature}`
}

// Consumer-side verification (constant-time compare)
function verifySignature(rawBody: string, header: string, secret: string, toleranceSec = 300) {
  const parts = Object.fromEntries(
    header.split(",").map((p) => p.split("=") as [string, string])
  )
  const timestamp = Number(parts.t)
  if (Math.abs(Date.now() / 1000 - timestamp) > toleranceSec) {
    throw new Error("Timestamp outside tolerance window")
  }
  const expected = crypto
    .createHmac("sha256", secret)
    .update(`${timestamp}.${rawBody}`)
    .digest("hex")
  const a = Buffer.from(expected)
  const b = Buffer.from(parts.v1)
  if (a.length !== b.length || !crypto.timingSafeEqual(a, b)) {
    throw new Error("Invalid signature")
  }
}

Two details separate a correct implementation from a broken one. First, the timestamp must be part of the signed content and the consumer must reject signatures outside a tolerance window, typically around five minutes, otherwise an attacker who captures one valid request can replay it indefinitely. Second, the comparison between the computed and received signature must run in constant time. A naive string comparison that returns as soon as it finds a mismatched character leaks timing information an attacker can exploit to guess the correct signature one byte at a time.

Retries with exponential backoff, not a fixed schedule

When a delivery attempt fails, whether from a timeout, a five hundred response, or a closed connection, the naive fix is to retry immediately a fixed number of times. That approach makes a bad situation worse: if the consumer's endpoint is down because it is overloaded, an immediate retry storm from every event you are trying to send only deepens the outage. The fix is exponential backoff, where each retry waits longer than the last, combined with a small amount of random jitter so that many queued deliveries do not all wake up and retry at the exact same instant.

That final row matters as much as the early ones. Retrying forever wastes resources on an endpoint that is permanently gone, such as a consumer who deleted their integration months ago. Once you exhaust your retry budget, move the delivery into a dead letter state, notify the consumer through a dashboard or email digest, and let them trigger a manual replay once they have fixed whatever was broken on their side.

Idempotency: making duplicates harmless

Because at-least-once delivery guarantees duplicates will happen, the consumer's handler needs to be idempotent, meaning processing the same event twice produces the same end state as processing it once. This is a job for both sides of the integration: you provide a stable event identifier, and the consumer stores which identifiers they have already processed. The pattern is straightforward to implement correctly.

1. Generate a unique, stable identifier for every event at creation time, before any delivery attempt, and never change it across retries.
2. Have the consumer check that identifier against a small table of processed events before doing any side-effecting work, such as charging a card or updating inventory.
3. Have the consumer record the identifier as processed in the same database transaction as the side effect itself, so a crash between the two steps cannot cause a duplicate to slip through.
4. Return a success response for an already-processed identifier rather than an error, since from your sender's perspective the event was in fact handled.

The event log: your source of truth for replay

Every event you ever intend to deliver should first be written to a durable event log, separate from the delivery attempt itself. This separation matters because it decouples the question of what happened in your system from the question of whether a particular consumer's endpoint successfully received it. With that separation, replay becomes a simple operation: insert a new delivery row pointing at an existing event and let your delivery worker pick it up on its normal schedule.

-- Event log table for replay support
CREATE TABLE webhook_events (
  id UUID PRIMARY KEY,
  event_type TEXT NOT NULL,
  payload JSONB NOT NULL,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE webhook_deliveries (
  id UUID PRIMARY KEY,
  event_id UUID NOT NULL REFERENCES webhook_events(id),
  endpoint_id UUID NOT NULL,
  status TEXT NOT NULL, -- pending | delivered | failed | dead_letter
  attempt_count INT NOT NULL DEFAULT 0,
  next_attempt_at TIMESTAMPTZ,
  last_response_code INT,
  UNIQUE (event_id, endpoint_id)
);

-- Replaying a single event to a single endpoint on demand
INSERT INTO webhook_deliveries (id, event_id, endpoint_id, status, next_attempt_at)
VALUES (gen_random_uuid(), $1, $2, 'pending', now())
ON CONFLICT (event_id, endpoint_id) DO UPDATE
  SET status = 'pending', next_attempt_at = now();

The event log also earns its keep well beyond error recovery. Consumers with a bug in an older version of their handler can request a replay of the last week's events after shipping a fix. New integrations that were only just configured can backfill their initial state from historical events instead of waiting for the next occurrence of each event type. Keep the log immutable, retain events for a bounded window that matches your compliance and storage needs, and index it by both event type and creation time so replays by category are cheap.

Operational guardrails worth building in from the start

A handful of smaller decisions determine whether a webhook system stays healthy as it scales past a handful of consumers to hundreds or thousands of registered endpoints.

• Enforce a strict request timeout, typically a few seconds, so one slow consumer cannot tie up delivery workers that other consumers depend on.
• Track a per-endpoint failure rate and automatically pause or flag endpoints that fail nearly every delivery, rather than silently retrying a dead integration forever.
• Version your payload schema explicitly and never remove a field consumers may already depend on; add new fields instead, and only introduce breaking changes behind a new payload version consumers opt into.

None of these four pillars, signing, backoff, idempotency, and a replayable log, are difficult in isolation. The difficulty is in treating them as a cohesive system from the start rather than bolting them on after the first production incident. Consumers judge a webhook API by how it behaves during their worst day, not their best one, and a design built around at-least-once delivery with the tooling to make duplicates and outages non-events is what earns their trust.