The Idempotent Consumer Pattern for Message Queues

Every message queue I have run in production, from RabbitMQ to Kafka to SQS, makes the same promise: at-least-once delivery. Not exactly-once. That word matters. It means the broker will redeliver a message if it is not sure you finished processing it, and it is frequently not sure. A consumer crashes after doing the work but before acknowledging. A network blip drops the ack. A visibility timeout expires while you were still busy. In every case the broker does the safe thing and hands you the message again.

So duplicates are not an edge case you might hit. They are a certainty you must design for. If your consumer charges a card, sends an email, or decrements inventory, processing the same message twice is a real bug with real money attached. The idempotent consumer pattern is the standard, boring, correct answer, and it lives almost entirely in your database.

Why exactly-once delivery is a myth

Exactly-once delivery across a network is impossible to guarantee, because the acknowledgement itself can be lost. If the consumer acks and the ack vanishes, the broker must choose: redeliver and risk a duplicate, or drop and risk losing the message forever. At-least-once picks the first, which is the only safe choice. What vendors market as exactly-once is really at-least-once delivery plus idempotent processing on the consumer side. Kafka's exactly-once semantics work inside the Kafka boundary; the moment you write to Postgres or call Stripe, you are back to needing your own idempotency.

The processed-messages table

The whole pattern rests on one table. Each row is the fingerprint of a message you have already handled. The message ID column carries a unique constraint, which is the load-bearing element of the entire design. When a duplicate arrives, the insert violates that constraint and fails loudly, and that failure is exactly the signal you want.

CREATE TABLE processed_messages (
    message_id   TEXT        NOT NULL,
    consumer     TEXT        NOT NULL,
    processed_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (message_id, consumer)
);

-- The composite key lets different consumers each
-- process the same message once, independently.

Two design notes. First, key on the business message ID, not the broker offset or the delivery tag. Offsets reset, tags are per-connection; a stable producer-assigned ID survives redelivery and rebalancing. Second, include the consumer name in the key when multiple distinct consumers subscribe to the same topic, so each gets its own once-only guarantee.

One transaction is the whole trick

The subtle mistake is to check the table, then process, then insert. That leaves a gap: two duplicate deliveries can both read empty, both process, both insert. The correct shape wraps the dedup insert and the business side effect in a single database transaction. If the insert succeeds, do the work and commit. If the insert hits the unique constraint, the message is a duplicate, so roll back and acknowledge it without doing anything. The atomicity is what makes it safe under concurrency.

// NestJS consumer, TypeORM data source
async function handle(msg: IncomingMessage): Promise<void> {
  await dataSource.transaction(async (tx) => {
    try {
      await tx.insert(ProcessedMessage, {
        messageId: msg.id,
        consumer: 'billing-service',
      });
    } catch (err) {
      if (isUniqueViolation(err)) {
        // Already processed. Roll back, ack, move on.
        return;
      }
      throw err; // real error -> nack -> redelivery
    }

    // Same transaction as the dedup row:
    await tx.update(Order, msg.orderId, { status: 'PAID' });
  });
}

// Postgres unique-violation SQLSTATE
function isUniqueViolation(e: unknown): boolean {
  return (e as { code?: string })?.code === '23505';
}

When the side effect leaves your database

A local row update rolls back cleanly. Charging a card through Stripe does not; once that HTTP call succeeds, the money moved, and no transaction rollback un-charges it. Here you push idempotency out to the third party. Most serious payment and messaging APIs accept an idempotency key, and the IETF has an active Standards Track draft for an Idempotency-Key HTTP header, revision 07 as of late 2025. Send a stable key derived from the message ID; the provider deduplicates on their side and returns the original result on retry.

• Derive the idempotency key deterministically from the message ID, so a redelivery produces the exact same key.
• Persist the external call's result in your transaction before committing, so you never repeat the call on the next redelivery.
• Order operations so the external effect is the last thing before commit, shrinking the window where a crash leaves work half-done.

Cleanup, growth, and the details that bite

The processed-messages table grows forever if you let it. You do not need rows older than your broker's maximum redelivery window, which is usually hours, not months. Prune on a schedule and the table stays small and its unique index stays fast.

-- Nightly cron; keep a generous safety margin over
-- the broker's redelivery / retention window.
DELETE FROM processed_messages
WHERE processed_at < now() - INTERVAL '7 days';

Put together, the pattern is unglamorous and that is the point. Assume at-least-once delivery, key on a stable message ID, insert into a table with a unique constraint inside the same transaction as your side effect, push idempotency to any external system, and prune old rows. That is how you build exactly-once effects on a broker that only promises at-least-once, and it is the same shape whether you run Kafka, RabbitMQ, or SQS.