# Event-Driven Architecture Explained: Queues, Logs and Outbox

> Event-driven architecture explained: events vs commands, queues vs logs, at-least-once delivery, idempotent consumers, ordering, backlog and the outbox.

URL: https://www.matthewswong.com/en/learn/system-design/event-driven-architecture-explained/

System design, step by step · Lesson 11 of 11 · Reviewed on 18 October 2026 · By Matthews Wong (https://www.matthewswong.com/en/about/)

> Event-driven architecture is a design where services communicate by publishing events, records that something happened, to a broker, and other services react on their own schedule. The producer never waits for its consumers. A queue gives each message to one worker and deletes it once acknowledged; a log keeps events, so each consumer group reads at its own pace. Delivery is usually at least once, so consumers must be idempotent.

## You will be able to

- Tell an event from a command, and choose between a queue and a log for a given flow.
- Explain why at-least-once delivery forces idempotent consumers, and keep events in order with a partition key.
- Publish events safely with the outbox pattern, and read consumer lag before a backlog turns into an outage.

## One event, three consumers, one duplicate

Checkout publishes one OrderPlaced event to a log; email and analytics read it at their own pace while stock is down, stock later catches up from its saved position, receives one event twice after a crash, and skips the duplicate because it records the ids of the events it has processed.

1. **Meet the parts.** Checkout publishes order events to a log. Email, stock and analytics each read it with their own consumer group and position. Stock records the event ids it has processed.
2. **Publish once.** Checkout appends one OrderPlaced event, evt-42, and moves on. It does not know who listens and waits for nobody: the log stores the event, and each reader fetches it later.

   ```
   XADD orders * event OrderPlaced id evt-42
   ```
3. **Each reads at its pace.** Email and analytics each read evt-42 at their own pace and move their own position forward. Stock is down, so its position stays put and the event waits in the log.

   ```
   XREADGROUP GROUP email email-1 STREAMS orders >
   ```

   *Predict:* Orders 43 and 44 arrive while stock is still down. When stock restarts, what does it read? (Only orders placed after it restarts / All three it missed, oldest first / Nothing, until checkout sends them again)
4. **Catch up from the backlog.** Orders 43 and 44 arrived meanwhile. Stock restarts at its saved position and reads its backlog, 42 to 44, in order. Its lag falls from 3 to 0.

   ```
   XINFO GROUPS orders
   ```
5. **Crash, then redelivery.** Stock reserves items for evt-44, then crashes before acknowledging it. The broker cannot tell whether the work happened, so evt-44 is delivered again: at-least-once delivery.

   ```
   XPENDING orders stock - + 10
   ```

   *Predict:* Stock already reserved the items for order 44. What prevents a second reservation? (Stock skips an event id it already processed / The broker never delivers an event twice / Checkout notices and cancels the duplicate)
6. **Skip the duplicate.** Stock finds evt-44 among its processed ids, saved in the same transaction as each reservation. So it skips the work and acknowledges. One extra delivery, no double reservation.

   ```
   INSERT INTO processed_events (event_id) VALUES ('evt-44')
   ON CONFLICT DO NOTHING;  -- 0 rows: already done
   ```

## What is event-driven architecture?

Event-driven architecture is a way of connecting services in which one service publishes an event, a record that something happened, and any number of others react to it. Kafka's documentation defines an event in exactly those terms, as a record that something happened in the world or in your business, and calls producers and consumers fully decoupled and agnostic of each other: a producer never waits for a consumer. Between them sits a broker that stores events and hands them out.

Compare it with the synchronous chain in lesson 6, where checkout calls stock, stock calls payment, and one slow hop slows every request. Here checkout writes OrderPlaced once and returns. Email, stock and analytics each subscribe, and a fourth service can subscribe next month without anyone touching checkout. Martin Fowler names the price beside the benefit: event notification implies a low level of coupling and is simple to set up, but a flow that runs across several events can be hard to see, because it is not explicit in any program text.

## What is the difference between an event and a command?

An event states a fact that already happened, so it is usually named in the past tense: OrderPlaced, PaymentFailed. Its producer is not asking anyone for anything and, in Fowler's words, does not really care much about the response. A command asks one particular receiver to do something: SendReceipt, ReserveStock. It has an owner who is expected to act, and that owner can refuse it.

Fowler warns about the event used as a passive-aggressive command: the sender expects the receiver to carry out an action and ought to send a command, but styles the message as an event instead. If checkout publishes OrderPlaced and then breaks when no receipt goes out, it was really sending a command, and that dependency deserves to be visible as one. Both kinds can travel through the same broker; what differs is who decides what happens next.

## Queue or log: what happens to a message after it is read?

A classic queue, as RabbitMQ implements it, is an ordered collection of messages waiting for consumers. Consumers on one queue compete: by default RabbitMQ sends each message to the next consumer in sequence, so adding workers drains the queue faster. Once a consumer acknowledges a delivery, the broker marks the message for deletion. A queue remembers the work still to do, not what happened.

A log keeps events after they are read. Kafka stores events in topics, does not delete them after consumption and discards them only after the topic's retention period. A consumer's progress in each partition is a single number, the offset of the next event to read, and a consumer can rewind to an old offset and read again. Every consumer group sees every event; inside a group, each partition is read by one consumer at a time. RabbitMQ streams and Redis Streams work the same way, and RabbitMQ calls it non-destructive consumer semantics.

So choose by what the message is. A job that must run once, such as rendering one invoice, fits a queue. A fact that several services read, that a new service may need to replay, or that you want to audit later, fits a log.

## Why does a consumer get the same event twice?

Because the alternative is losing events. Kafka's design documentation lists three guarantees: at most once, where messages may be lost but are never redelivered; at least once, where they are never lost but may be redelivered; and exactly once. Kafka gives at least once by default. The reason is ordinary: a consumer that reads, processes and then saves its position will process again whatever it finished just before a crash. Saving the position first gives at most once instead, and loses that work.

RabbitMQ behaves the same way. A delivery that was still unacknowledged when its channel or connection closed is requeued and delivered again with the redeliver flag set, possibly to another consumer, and the documentation tells consumers to be prepared for redeliveries. Kafka can reach exactly once between Kafka topics with transactions, but its documentation adds that exactly-once delivery into other systems generally requires their cooperation. Your stock table and your email provider are other systems.

So make the consumer idempotent, the property lesson 10 teaches for APIs. The idempotent consumer pattern records the id of every processed message in the same database transaction as the business change; if inserting the id fails because it already exists, the message was handled before and the consumer skips it. Kafka's documentation describes the same trick for positions: let the consumer store its offset in the same place as its output.

## How do partitions decide the order of events?

Kafka spreads a topic over partitions so that many consumers can read in parallel. Events with the same key, such as a customer or order id, are written to the same partition, and any consumer of a partition reads its events in exactly the order they were written. That guarantee is per partition. Two events with different keys can sit in different partitions, which are read independently.

Pick the key by asking which events must stay in order. For an order's life, OrderPlaced, OrderPaid, OrderCancelled, the key is the order id. If OrderPlaced is keyed by the order and OrderCancelled by something else, they can land in different partitions and a consumer may see the cancellation first. Competing consumers on one queue have the same gap: RabbitMQ's documentation notes that any redelivery can change the order, and Redis's warns that with one stream and several consumers, messages about the same item may be processed out of order.

## What happens when consumers fall behind?

Producers and consumers run at different speeds, and the gap between them is the backlog. In a log it has a name, lag: Redis reports a consumer group's lag as the number of entries still waiting to be delivered to it, and suggests adding consumers when it stays high. Kafka's consumers pull, which its design documentation calls the nicer property: a consumer that falls behind simply catches up when it can. Lag that rises during a burst and falls afterwards is healthy; lag that only rises is the number to alert on.

Backpressure is how a slow consumer limits what it receives. RabbitMQ's prefetch caps the number of unacknowledged deliveries on a channel; when the cap is reached, the broker stops delivering on that channel until one is acknowledged. That protects the consumer, not the broker, because the backlog still grows in the queue. A length limit stops that, but read what it does when full: RabbitMQ's default overflow drops or dead-letters messages from the front of the queue, the oldest first, while reject-publish refuses new ones and tells a publisher that uses confirms with a nack, pushing the problem back to the producer.

A log has a limit too, and it is time. Kafka discards old events once a topic's retention period has passed, so a consumer that is down for longer than that period loses the events it never read. Redis streams capped with MAXLEN trim old entries the same way, and a group whose unread entries were trimmed reports its lag as unknown.

## How do you publish an event without losing it?

The trap is the dual write. Checkout must save the order in its database and publish OrderPlaced to the broker: two systems with no shared transaction. Commit first and crash before publishing, and the order exists with nobody told. Publish first and let the transaction roll back, and email, stock and analytics act on an order that never existed. microservices.io states the rule both ways: if the transaction commits, the message must be sent; if it rolls back, it must not. Two-phase commit is often not an option, because the database or the broker may not support it.

The transactional outbox makes the event part of the same local transaction: insert the order row and an outbox row together, then let a separate relay read the outbox and publish. The relay either polls the table, a polling publisher, or tails the database's transaction log. It can publish a row twice, for instance when it crashes after sending and before marking the row as sent, so consumers must be idempotent here too. The pattern also requires that messages reach the broker in the order the service wrote them.

## Where do CQRS and event sourcing fit?

They are separate ideas that often travel with events. Event sourcing stores the events themselves as the record of truth, so that, as Fowler puts it, you can confidently rebuild the system state by reprocessing the events at any time in the future. CQRS is the notion of having separate data structures for reading and writing, and an event stream is one way to keep the read side up to date. A lighter step is event-carried state transfer, where events carry the data so consumers keep their own copy and stop calling the producer back: recipients keep working when the source is unavailable, at the cost of many copies and more complexity on the receiver. Fowler's caution about CQRS, that the gain has to be balanced against the additional complexity of separate models, fits the other two as well. You can publish events without adopting any of them.

## When should you not go event-driven?

When the caller needs the answer now. Checking a password, showing whether an item is in stock before the Buy button: these are queries, and a request with a response is simpler to build and to debug. When one database can hold the data, a local transaction beats an outbox, a broker and idempotent consumers. And when the flow is a process whose steps must happen in sequence, keep the sequence visible in one place, an orchestrator or a plain function, rather than spreading it across services as the chain of events Fowler says is hard to see.

Events earn their place when several parties must react to one fact, when the producer must not wait for them, when a consumer may be down for a while without the producer noticing, or when a burst has to be absorbed and worked off later. A broker is one more system to run, watch and upgrade, so start with the smallest one that gives you the guarantees above.

## The real config behind this lesson

`.github/workflows/deploy.yml:7-10`

```yaml
on:
  push:
    branches: [main]
  workflow_dispatch:
```

A push is an event: the fact that commits landed on a branch. GitHub then starts a run of every workflow whose on: values match, so this file is one subscriber among possibly many, and branches: [main] is its filter. workflow_dispatch is the other kind of message: a person pressing Run workflow asks this one workflow to act, which is a command.

`.github/workflows/deploy.yml:12-18`

```yaml
# Concurrency is split per job rather than set once for the whole workflow.
#
# A single workflow-level group with cancel-in-progress: false made a burst of
# commits queue a burst of complete pipelines — 28 deep on 2026-09-24, 27 of
# them building code that was already superseded. Flipping that one group to
# true would have fixed the queue and allowed a run to be cancelled MID-SSH
# DEPLOY, which is the one place here that must never be interrupted.
```

Every push is its own event, and every one started a complete pipeline. The comment is this repo's own record of a burst on 2026-09-24: 28 pipelines deep, 27 of them building code a newer commit had already replaced. That is a backlog in miniature: the consumer was slower than the producer, and nothing told it which waiting events no longer mattered. GitHub's documented default is narrower than a queue: a concurrency group holds at most one waiting run, and a newer one cancels it.

`.github/workflows/deploy.yml:28-31`

```yaml
    # Superseded verification is pure waste; drop it the moment it is stale.
    concurrency:
      group: build-${{ github.ref }}
      cancel-in-progress: true
```

The fix tells the consumer which events no longer matter. The group key is the Git ref, so pushes to the same branch share one group, much as events with one key share a partition, and cancel-in-progress: true cancels the running build when a newer one joins. It behaves like Kafka's log compaction, which keeps at least the last value for each key. That is safe only because a build of the newest commit includes every older push; events that each carry a separate change, like payments, can never be dropped this way.

`.github/workflows/deploy.yml:297-301`

```yaml
    # Two runs racing would each diff against the same snapshot and the slower
    # one would overwrite the other's; queue them instead.
    concurrency:
      group: indexnow
      cancel-in-progress: false
```

The IndexNow job is a consumer with memory: each run compares the live sitemap with the snapshot the previous run saved. Two runs at once would read the same snapshot and the slower would overwrite the other's result, so its group never cancels a running job: a new run waits until the current one ends. By default at most one run waits and a newer arrival replaces it, which is safe here because the newest run compares against the newest snapshot. A Kafka partition is read by one consumer per group for the same reason: when handling an event depends on the one before it, one reader at a time keeps the order.

## What breaks

- **Saving and publishing in two separate steps.** Commit the order and then publish, and a crash in between loses the event for good; publish first, and a rollback announces an order that never existed. Neither order is safe, and a retry cannot help a process that already died. Write the event to an outbox table in the same transaction and let a relay publish it, with idempotent consumers for the duplicates it may send.
- **A poison message that blocks everything behind it.** An event that always crashes its consumer is delivered again and crashes it again. In a log, the group's position cannot move past it, so every event behind it in that partition waits. Count deliveries and set the message aside after a limit: RabbitMQ quorum queues default to 20 since 4.0, then drop or dead-letter it, and Redis Streams exposes a delivery counter for the same job. Alert on whatever lands in the dead-letter place.
- **A backlog nobody watches.** When consumers are slower than producers, the backlog grows quietly until something gives: memory, disk, or a retention period that discards events a slow group never read. A full RabbitMQ queue with a length limit drops or dead-letters its oldest messages by default. Alert on lag that keeps growing, add consumers, and decide on purpose whether a full queue should drop, dead-letter or refuse new messages.
- **Expecting order across partitions.** OrderPlaced and OrderCancelled for one order, published with different keys, can land in different partitions and be read cancellation first. Kafka's ordering guarantee is per partition. Key every event about an entity by that entity's id, and let a consumer park an event it cannot apply yet instead of failing on it.

### Queue or log: what each one does with a message after it is read

|  | Queue (RabbitMQ-style) | Log (Kafka-style) |
| --- | --- | --- |
| After an acknowledgement | The message is marked for deletion | The event stays until retention expires |
| Several consumers | Compete: each message goes to one of them | Every group sees every event; one consumer per partition inside a group |
| Progress is tracked as | Each delivery waiting for its ack | One offset per partition per group |
| Replay | Not possible once acknowledged | Rewind the offset and read again |
| Order | First in, first out, until a redelivery or a priority reorders it | Guaranteed within a partition only |
| Fits | Jobs that must run once, such as rendering an invoice | Facts that many services read, replay or audit |

## Try it yourself

You need Docker and nothing else. Redis Streams is a small log with consumer groups, enough to see fan-out, a backlog, a redelivery and a replay on a laptop. Run every step in one terminal, because step 2 keeps the event ids in shell variables.

1. Start Redis and create the orders stream with one consumer group per service, email and stock. Each group keeps its own position, starting at the end of the stream.

   ```bash
   docker run -d --name eda-redis redis:7-alpine
   docker exec eda-redis redis-cli XGROUP CREATE orders email '$' MKSTREAM
   docker exec eda-redis redis-cli XGROUP CREATE orders stock '$'
   ```
2. Publish three OrderPlaced events, once each. XADD prints each entry's id; keep them in variables for the acknowledgements later.

   ```bash
   A=$(docker exec eda-redis redis-cli XADD orders '*' event OrderPlaced id evt-41)
   B=$(docker exec eda-redis redis-cli XADD orders '*' event OrderPlaced id evt-42)
   C=$(docker exec eda-redis redis-cli XADD orders '*' event OrderPlaced id evt-43)
   echo $A $B $C
   ```
3. Email reads all three and acknowledges them; stock is down and reads nothing. XINFO GROUPS shows email at pending 0 and lag 0, and stock at lag 3: its backlog waits in the stream.

   ```bash
   docker exec eda-redis redis-cli XREADGROUP GROUP email email-1 COUNT 10 STREAMS orders '>'
   docker exec eda-redis redis-cli XACK orders email $A $B $C
   docker exec eda-redis redis-cli XINFO GROUPS orders
   ```
4. Stock comes back and reads its backlog, then pretend it crashed before acknowledging. On restart it reads its own pending history with id 0 and gets the same three events again. XPENDING shows each one delivered twice.

   ```bash
   docker exec eda-redis redis-cli XREADGROUP GROUP stock stock-1 COUNT 10 STREAMS orders '>'
   docker exec eda-redis redis-cli XREADGROUP GROUP stock stock-1 COUNT 10 STREAMS orders 0
   docker exec eda-redis redis-cli XPENDING orders stock - + 10
   ```
5. Make the consumer idempotent by remembering each event id it handled. The first SET NX prints OK; the second prints an empty reply, so the duplicate is skipped. Then acknowledge. In a real service, store the id in the same database transaction as the stock change.

   ```bash
   docker exec eda-redis redis-cli SET seen:stock:evt-41 1 NX
   docker exec eda-redis redis-cli SET seen:stock:evt-41 1 NX
   docker exec eda-redis redis-cli XACK orders stock $A $B $C
   ```
6. Replay: a new analytics service starts its group at 0 and reads all three events, although email and stock acknowledged them already. XLEN still counts 3, because a log keeps what was read. A queue would have nothing left to give it.

   ```bash
   docker exec eda-redis redis-cli XGROUP CREATE orders analytics 0
   docker exec eda-redis redis-cli XREADGROUP GROUP analytics analytics-1 COUNT 10 STREAMS orders '>'
   docker exec eda-redis redis-cli XLEN orders
   ```
7. Clean up the container.

   ```bash
   docker rm -f eda-redis
   ```

## Quick quiz

1. **Email, stock and analytics must each react to every OrderPlaced event, and a fraud service added next month must read the last seven days. Which fits?**
   A. One queue that all four services consume from
   B. A log retained for at least seven days, one consumer group per service
   C. Checkout calls each service's API after every order
   Answer: B. A shared queue gives each message to one consumer and deletes it once acknowledged, so the services would split the events and the fraud service would find nothing old. A log keeps events for its retention period, and every group reads all of them from its own position.

2. **A stock consumer reserves items, then commits its offset. It crashes between the two. What happens after it restarts?**
   A. The event is skipped, because it was already delivered
   B. Nothing, because Kafka delivers exactly once by default
   C. The event is delivered again, so the reservation must be idempotent
   D. The producer publishes the event again
   Answer: C. Reading, processing and then saving the position is at-least-once: the saved offset still points at the event, so it is read again. Kafka's default is at least once, so the consumer must recognise an event id it already handled.

3. **OrderPlaced is keyed by order id and OrderCancelled by customer id, on a topic with six partitions. What can a consumer group see?**
   A. Sometimes the cancellation before the order
   B. Always the order first, because the topic keeps write order
   C. An error, because one order's events span two partitions
   Answer: A. Kafka guarantees order within a partition. Different keys can land in different partitions, which are read independently, so the cancellation may be read first. Key every event about an order by the order id.

## Think it through

- **Is SendReceipt an event or a command, and what about ReceiptSent?** SendReceipt is a command: it asks one receiver to act, and that receiver can refuse. ReceiptSent is an event: a fact that already happened, which any number of services may react to or ignore.
- **Why can a new service replay last week's orders from a log but not from a queue?** A queue deletes a message once it is acknowledged. A log keeps events until its retention expires and tracks each group's progress as an offset, so a new group can start at the beginning.
- **Checkout saves an order and then publishes OrderPlaced. What can go wrong, and what fixes it?** A crash between the two leaves an order nobody hears about, and publishing first risks announcing an order that rolls back. Write the event to an outbox table in the same transaction and let a relay publish it, with idempotent consumers for any duplicates.
- **What does consumer lag measure, and when should it worry you?** The number of events written but not yet delivered to a group. Lag that rises during a burst and falls afterwards is normal. Lag that keeps growing means consumers are slower than producers, and with time-based retention, unread events will eventually be discarded.

## Go deeper

- [Event-Driven Architecture with Kafka and NestJS: A Production Guide](https://www.matthewswong.com/en/blog/event-driven-kafka-nestjs/)
- [The Transactional Outbox Pattern with PostgreSQL](https://www.matthewswong.com/en/blog/transactional-outbox-pattern-postgres/)
- [CQRS and Event Sourcing with NestJS: When Complexity is Worth It](https://www.matthewswong.com/en/blog/cqrs-event-sourcing-nestjs/)
- [NATS JetStream vs Kafka for Teams That Do Not Need Kafka's Scale](https://www.matthewswong.com/en/blog/nats-jetstream-vs-kafka-lightweight/)

## Sources

- [Apache Kafka 4.3 docs: Introduction (events, topics, partitions, retention)](https://kafka.apache.org/43/getting-started/introduction/)
- [Apache Kafka 4.3 docs: Design (delivery semantics, offsets, compaction)](https://kafka.apache.org/43/design/design/)
- [RabbitMQ docs: Queues](https://www.rabbitmq.com/docs/queues)
- [RabbitMQ docs: Consumer acknowledgements and publisher confirms](https://www.rabbitmq.com/docs/confirms)
- [RabbitMQ tutorial: Work queues (round-robin dispatch)](https://www.rabbitmq.com/tutorials/tutorial-two-python)
- [RabbitMQ docs: Streams (non-destructive consumers, replay)](https://www.rabbitmq.com/docs/streams)
- [RabbitMQ docs: Quorum queues (poison message handling)](https://www.rabbitmq.com/docs/quorum-queues)
- [RabbitMQ docs: Queue length limit and overflow behaviour](https://www.rabbitmq.com/docs/maxlength)
- [Redis docs: Streams and consumer groups](https://redis.io/docs/latest/develop/data-types/streams/)
- [Redis docs: XINFO GROUPS (consumer group lag)](https://redis.io/docs/latest/commands/xinfo-groups/)
- [Martin Fowler: What do you mean by “Event-Driven”?](https://www.martinfowler.com/articles/201701-event-driven.html)
- [microservices.io: Transactional outbox](https://microservices.io/patterns/data/transactional-outbox.html)
- [microservices.io: Idempotent consumer](https://microservices.io/patterns/communication-style/idempotent-consumer.html)
- [GitHub Docs: Workflows (how an event triggers runs)](https://docs.github.com/en/actions/concepts/workflows-and-actions/workflows)
- [GitHub Docs: Events that trigger workflows (push, workflow_dispatch)](https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows)
- [GitHub Docs: Control the concurrency of workflows and jobs](https://docs.github.com/en/actions/how-tos/write-workflows/choose-when-workflows-run/control-workflow-concurrency)

---

- HTML version: https://www.matthewswong.com/en/learn/system-design/event-driven-architecture-explained/
- Bahasa Indonesia: https://www.matthewswong.com/id/learn/system-design/event-driven-architecture-explained/
- Site index for agents: https://www.matthewswong.com/llms.txt
- Sitemap: https://www.matthewswong.com/sitemap.xml
- API description (OpenAPI): https://www.matthewswong.com/openapi.json