System design, step by step · Lesson 11 of 11 · 15 min

Event-Driven Architecture Explained: Queues, Logs and Outbox

Event-driven architecture explained: events vs commands, queues vs logs, at-least-once delivery, idempotent consumers, ordering, backlog and the outbox.

By Matthews WongReviewed on 18 October 2026

Key takeaway

Event-driven architecture is a design where services communicate by publishing events, records that something happened, to a broker, and other services react on their own schedule. The producer never waits for its consumers. A queue gives each message to one worker and deletes it once acknowledged; a log keeps events, so each consumer group reads at its own pace. Delivery is usually at least once, so consumers must be idempotent.

You will be able to

Tell an event from a command, and choose between a queue and a log for a given flow.
Explain why at-least-once delivery forces idempotent consumers, and keep events in order with a partition key.
Publish events safely with the outbox pattern, and read consumer lag before a backlog turns into an outage.

Watch it work

Step through the diagram

Press Play, or step with the arrows. Every step is written out below the diagram.

One event, three consumers, one duplicate

Checkout publishes one OrderPlaced event to a log; email and analytics read it at their own pace while stock is down, stock later catches up from its saved position, receives one event twice after a crash, and skips the duplicate because it records the ids of the events it has processed.

Checkout publishes order events to a log. Email, stock and analytics each read it with their own consumer group and position. Stock records the event ids it has processed.

Keys: ← → step · Home / End · K play or pause

Checkout publishes order events to a log. Email, stock and analytics each read it with their own consumer group and position. Stock records the event ids it has processed.
Checkout appends one OrderPlaced event, evt-42, and moves on. It does not know who listens and waits for nobody: the log stores the event, and each reader fetches it later.
XADD orders * event OrderPlaced id evt-42
Email and analytics each read evt-42 at their own pace and move their own position forward. Stock is down, so its position stays put and the event waits in the log.
XREADGROUP GROUP email email-1 STREAMS orders >Predict the next step Orders 43 and 44 arrive while stock is still down. When stock restarts, what does it read? (Only orders placed after it restarts · All three it missed, oldest first · Nothing, until checkout sends them again)
Orders 43 and 44 arrived meanwhile. Stock restarts at its saved position and reads its backlog, 42 to 44, in order. Its lag falls from 3 to 0.
XINFO GROUPS orders
Stock reserves items for evt-44, then crashes before acknowledging it. The broker cannot tell whether the work happened, so evt-44 is delivered again: at-least-once delivery.
XPENDING orders stock - + 10Predict the next step Stock already reserved the items for order 44. What prevents a second reservation? (Stock skips an event id it already processed · The broker never delivers an event twice · Checkout notices and cancels the duplicate)
Stock finds evt-44 among its processed ids, saved in the same transaction as each reservation. So it skips the work and acknowledges. One extra delivery, no double reservation.
INSERT INTO processed_events (event_id) VALUES ('evt-44') ON CONFLICT DO NOTHING; -- 0 rows: already done

What is event-driven architecture?

Event-driven architecture is a way of connecting services in which one service publishes an event, a record that something happened, and any number of others react to it. Kafka's documentation defines an event in exactly those terms, as a record that something happened in the world or in your business, and calls producers and consumers fully decoupled and agnostic of each other: a producer never waits for a consumer. Between them sits a broker that stores events and hands them out.

Read more (1)

Compare it with the synchronous chain in lesson 6, where checkout calls stock, stock calls payment, and one slow hop slows every request. Here checkout writes OrderPlaced once and returns. Email, stock and analytics each subscribe, and a fourth service can subscribe next month without anyone touching checkout. Martin Fowler names the price beside the benefit: event notification implies a low level of coupling and is simple to set up, but a flow that runs across several events can be hard to see, because it is not explicit in any program text.

What is the difference between an event and a command?

An event states a fact that already happened, so it is usually named in the past tense: OrderPlaced, PaymentFailed. Its producer is not asking anyone for anything and, in Fowler's words, does not really care much about the response. A command asks one particular receiver to do something: SendReceipt, ReserveStock. It has an owner who is expected to act, and that owner can refuse it.

Read more (1)

Fowler warns about the event used as a passive-aggressive command: the sender expects the receiver to carry out an action and ought to send a command, but styles the message as an event instead. If checkout publishes OrderPlaced and then breaks when no receipt goes out, it was really sending a command, and that dependency deserves to be visible as one. Both kinds can travel through the same broker; what differs is who decides what happens next.

Queue or log: what happens to a message after it is read?

A classic queue, as RabbitMQ implements it, is an ordered collection of messages waiting for consumers. Consumers on one queue compete: by default RabbitMQ sends each message to the next consumer in sequence, so adding workers drains the queue faster. Once a consumer acknowledges a delivery, the broker marks the message for deletion. A queue remembers the work still to do, not what happened.

Why does a consumer get the same event twice?

Because the alternative is losing events. Kafka's design documentation lists three guarantees: at most once, where messages may be lost but are never redelivered; at least once, where they are never lost but may be redelivered; and exactly once. Kafka gives at least once by default. The reason is ordinary: a consumer that reads, processes and then saves its position will process again whatever it finished just before a crash. Saving the position first gives at most once instead, and loses that work.

How do partitions decide the order of events?

Kafka spreads a topic over partitions so that many consumers can read in parallel. Events with the same key, such as a customer or order id, are written to the same partition, and any consumer of a partition reads its events in exactly the order they were written. That guarantee is per partition. Two events with different keys can sit in different partitions, which are read independently.

Read more (1)

Pick the key by asking which events must stay in order. For an order's life, OrderPlaced, OrderPaid, OrderCancelled, the key is the order id. If OrderPlaced is keyed by the order and OrderCancelled by something else, they can land in different partitions and a consumer may see the cancellation first. Competing consumers on one queue have the same gap: RabbitMQ's documentation notes that any redelivery can change the order, and Redis's warns that with one stream and several consumers, messages about the same item may be processed out of order.

What happens when consumers fall behind?

Producers and consumers run at different speeds, and the gap between them is the backlog. In a log it has a name, lag: Redis reports a consumer group's lag as the number of entries still waiting to be delivered to it, and suggests adding consumers when it stays high. Kafka's consumers pull, which its design documentation calls the nicer property: a consumer that falls behind simply catches up when it can. Lag that rises during a burst and falls afterwards is healthy; lag that only rises is the number to alert on.

How do you publish an event without losing it?

The trap is the dual write. Checkout must save the order in its database and publish OrderPlaced to the broker: two systems with no shared transaction. Commit first and crash before publishing, and the order exists with nobody told. Publish first and let the transaction roll back, and email, stock and analytics act on an order that never existed. microservices.io states the rule both ways: if the transaction commits, the message must be sent; if it rolls back, it must not. Two-phase commit is often not an option, because the database or the broker may not support it.

Read more (1)

The transactional outbox makes the event part of the same local transaction: insert the order row and an outbox row together, then let a separate relay read the outbox and publish. The relay either polls the table, a polling publisher, or tails the database's transaction log. It can publish a row twice, for instance when it crashes after sending and before marking the row as sent, so consumers must be idempotent here too. The pattern also requires that messages reach the broker in the order the service wrote them.

Where do CQRS and event sourcing fit?

They are separate ideas that often travel with events. Event sourcing stores the events themselves as the record of truth, so that, as Fowler puts it, you can confidently rebuild the system state by reprocessing the events at any time in the future. CQRS is the notion of having separate data structures for reading and writing, and an event stream is one way to keep the read side up to date. A lighter step is event-carried state transfer, where events carry the data so consumers keep their own copy and stop calling the producer back: recipients keep working when the source is unavailable, at the cost of many copies and more complexity on the receiver. Fowler's caution about CQRS, that the gain has to be balanced against the additional complexity of separate models, fits the other two as well. You can publish events without adopting any of them.

When should you not go event-driven?

When the caller needs the answer now. Checking a password, showing whether an item is in stock before the Buy button: these are queries, and a request with a response is simpler to build and to debug. When one database can hold the data, a local transaction beats an outbox, a broker and idempotent consumers. And when the flow is a process whose steps must happen in sequence, keep the sequence visible in one place, an orchestrator or a plain function, rather than spreading it across services as the chain of events Fowler says is hard to see.

Read more (1)

Events earn their place when several parties must react to one fact, when the producer must not wait for them, when a consumer may be down for a while without the producer noticing, or when a burst has to be absorbed and worked off later. A broker is one more system to run, watch and upgrade, so start with the smallest one that gives you the guarantees above.

How this site does it

The real config behind this lesson

Excerpts from the repository that builds and deploys this website, quoted as they are.

.github/workflows/deploy.yml:7-10

on:
  push:
    branches: [main]
  workflow_dispatch:

A push is an event: the fact that commits landed on a branch. GitHub then starts a run of every workflow whose on: values match, so this file is one subscriber among possibly many, and branches: [main] is its filter. workflow_dispatch is the other kind of message: a person pressing Run workflow asks this one workflow to act, which is a command.

.github/workflows/deploy.yml:12-18

# Concurrency is split per job rather than set once for the whole workflow.
#
# A single workflow-level group with cancel-in-progress: false made a burst of
# commits queue a burst of complete pipelines — 28 deep on 2026-09-24, 27 of
# them building code that was already superseded. Flipping that one group to
# true would have fixed the queue and allowed a run to be cancelled MID-SSH
# DEPLOY, which is the one place here that must never be interrupted.

Every push is its own event, and every one started a complete pipeline. The comment is this repo's own record of a burst on 2026-09-24: 28 pipelines deep, 27 of them building code a newer commit had already replaced. That is a backlog in miniature: the consumer was slower than the producer, and nothing told it which waiting events no longer mattered. GitHub's documented default is narrower than a queue: a concurrency group holds at most one waiting run, and a newer one cancels it.

.github/workflows/deploy.yml:28-31

    # Superseded verification is pure waste; drop it the moment it is stale.
    concurrency:
      group: build-${{ github.ref }}
      cancel-in-progress: true

The fix tells the consumer which events no longer matter. The group key is the Git ref, so pushes to the same branch share one group, much as events with one key share a partition, and cancel-in-progress: true cancels the running build when a newer one joins. It behaves like Kafka's log compaction, which keeps at least the last value for each key. That is safe only because a build of the newest commit includes every older push; events that each carry a separate change, like payments, can never be dropped this way.

.github/workflows/deploy.yml:297-301

    # Two runs racing would each diff against the same snapshot and the slower
    # one would overwrite the other's; queue them instead.
    concurrency:
      group: indexnow
      cancel-in-progress: false

The IndexNow job is a consumer with memory: each run compares the live sitemap with the snapshot the previous run saved. Two runs at once would read the same snapshot and the slower would overwrite the other's result, so its group never cancels a running job: a new run waits until the current one ends. By default at most one run waits and a newer arrival replaces it, which is safe here because the newest run compares against the newest snapshot. A Kafka partition is read by one consumer per group for the same reason: when handling an event depends on the one before it, one reader at a time keeps the order.

What breaks

Mistakes that cost real time

Saving and publishing in two separate steps
Commit the order and then publish, and a crash in between loses the event for good; publish first, and a rollback announces an order that never existed. Neither order is safe, and a retry cannot help a process that already died. Write the event to an outbox table in the same transaction and let a relay publish it, with idempotent consumers for the duplicates it may send.
A poison message that blocks everything behind it
An event that always crashes its consumer is delivered again and crashes it again. In a log, the group's position cannot move past it, so every event behind it in that partition waits. Count deliveries and set the message aside after a limit: RabbitMQ quorum queues default to 20 since 4.0, then drop or dead-letter it, and Redis Streams exposes a delivery counter for the same job. Alert on whatever lands in the dead-letter place.
A backlog nobody watches
When consumers are slower than producers, the backlog grows quietly until something gives: memory, disk, or a retention period that discards events a slow group never read. A full RabbitMQ queue with a length limit drops or dead-letters its oldest messages by default. Alert on lag that keeps growing, add consumers, and decide on purpose whether a full queue should drop, dead-letter or refuse new messages.
Expecting order across partitions
OrderPlaced and OrderCancelled for one order, published with different keys, can land in different partitions and be read cancellation first. Kafka's ordering guarantee is per partition. Key every event about an entity by that entity's id, and let a consumer park an event it cannot apply yet instead of failing on it.

Queue or log: what each one does with a message after it is read
	Queue (RabbitMQ-style)	Log (Kafka-style)
After an acknowledgement	The message is marked for deletion	The event stays until retention expires
Several consumers	Compete: each message goes to one of them	Every group sees every event; one consumer per partition inside a group
Progress is tracked as	Each delivery waiting for its ack	One offset per partition per group
Replay	Not possible once acknowledged	Rewind the offset and read again
Order	First in, first out, until a redelivery or a priority reorders it	Guaranteed within a partition only
Fits	Jobs that must run once, such as rendering an invoice	Facts that many services read, replay or audit

Try it yourself

Do it on your own machine

You need Docker and nothing else. Redis Streams is a small log with consumer groups, enough to see fan-out, a backlog, a redelivery and a replay on a laptop. Run every step in one terminal, because step 2 keeps the event ids in shell variables.

Start Redis and create the orders stream with one consumer group per service, email and stock. Each group keeps its own position, starting at the end of the stream.

docker run -d --name eda-redis redis:7-alpine
docker exec eda-redis redis-cli XGROUP CREATE orders email '$' MKSTREAM
docker exec eda-redis redis-cli XGROUP CREATE orders stock '$'

Publish three OrderPlaced events, once each. XADD prints each entry's id; keep them in variables for the acknowledgements later.

A=$(docker exec eda-redis redis-cli XADD orders '*' event OrderPlaced id evt-41)
B=$(docker exec eda-redis redis-cli XADD orders '*' event OrderPlaced id evt-42)
C=$(docker exec eda-redis redis-cli XADD orders '*' event OrderPlaced id evt-43)
echo $A $B $C

Email reads all three and acknowledges them; stock is down and reads nothing. XINFO GROUPS shows email at pending 0 and lag 0, and stock at lag 3: its backlog waits in the stream.

docker exec eda-redis redis-cli XREADGROUP GROUP email email-1 COUNT 10 STREAMS orders '>'
docker exec eda-redis redis-cli XACK orders email $A $B $C
docker exec eda-redis redis-cli XINFO GROUPS orders

Stock comes back and reads its backlog, then pretend it crashed before acknowledging. On restart it reads its own pending history with id 0 and gets the same three events again. XPENDING shows each one delivered twice.

docker exec eda-redis redis-cli XREADGROUP GROUP stock stock-1 COUNT 10 STREAMS orders '>'
docker exec eda-redis redis-cli XREADGROUP GROUP stock stock-1 COUNT 10 STREAMS orders 0
docker exec eda-redis redis-cli XPENDING orders stock - + 10

5
Make the consumer idempotent by remembering each event id it handled. The first SET NX prints OK; the second prints an empty reply, so the duplicate is skipped. Then acknowledge. In a real service, store the id in the same database transaction as the stock change.
```
docker exec eda-redis redis-cli SET seen:stock:evt-41 1 NX
docker exec eda-redis redis-cli SET seen:stock:evt-41 1 NX
docker exec eda-redis redis-cli XACK orders stock $A $B $C
```
6
Replay: a new analytics service starts its group at 0 and reads all three events, although email and stock acknowledged them already. XLEN still counts 3, because a log keeps what was read. A queue would have nothing left to give it.
```
docker exec eda-redis redis-cli XGROUP CREATE orders analytics 0
docker exec eda-redis redis-cli XREADGROUP GROUP analytics analytics-1 COUNT 10 STREAMS orders '>'
docker exec eda-redis redis-cli XLEN orders
```
7
Clean up the container.
```
docker rm -f eda-redis
```

Quick quiz

Three questions before you move on

Pick an answer to see why it is right or wrong. Nothing is graded, so guess freely; a miss is kept in this browser so the course can ask it again later.

1.Email, stock and analytics must each react to every OrderPlaced event, and a fraud service added next month must read the last seven days. Which fits?
2.A stock consumer reserves items, then commits its offset. It crashes between the two. What happens after it restarts?
3.OrderPlaced is keyed by order id and OrderCancelled by customer id, on a topic with six partitions. What can a consumer group see?

From earlier lessons

Two questions from earlier lessons

Recalling an older lesson after a gap is what makes it stick. Answer from memory before you look anything up.

1.A POST with idempotency key k1 times out, and the client retries. When should it generate a new key?
2.Your API answers a missing order with status 200 and an error message in the body. What goes wrong?

From: 10. Idempotency Explained: Safe Retries and Idempotency Keys · 8. API Design for Beginners: Resources, Errors and Versioning

Think it through

Answer in your head, then check

1.Is SendReceipt an event or a command, and what about ReceiptSent?Show the answer
Answer:SendReceipt is a command: it asks one receiver to act, and that receiver can refuse. ReceiptSent is an event: a fact that already happened, which any number of services may react to or ignore.
2.Why can a new service replay last week's orders from a log but not from a queue?Show the answer
Answer:A queue deletes a message once it is acknowledged. A log keeps events until its retention expires and tracks each group's progress as an offset, so a new group can start at the beginning.
3.Checkout saves an order and then publishes OrderPlaced. What can go wrong, and what fixes it?Show the answer
Answer:A crash between the two leaves an order nobody hears about, and publishing first risks announcing an order that rolls back. Write the event to an outbox table in the same transaction and let a relay publish it, with idempotent consumers for any duplicates.
4.What does consumer lag measure, and when should it worry you?Show the answer
Answer:The number of events written but not yet delivered to a group. Lag that rises during a burst and falls afterwards is normal. Lag that keeps growing means consumers are slower than producers, and with time-based retention, unread events will eventually be discarded.

Go deeper

Related deep dives on this site

Sources

Primary sources checked on the research date. Follow them for the current version.

Previous lessonIdempotency Explained: Safe Retries and Idempotency Keys

Step through the diagram

One event, three consumers, one duplicate

What is event-driven architecture?

What is the difference between an event and a command?

Queue or log: what happens to a message after it is read?

Why does a consumer get the same event twice?

How do partitions decide the order of events?

What happens when consumers fall behind?

How do you publish an event without losing it?

Where do CQRS and event sourcing fit?

When should you not go event-driven?

The real config behind this lesson

Mistakes that cost real time

Saving and publishing in two separate steps

A poison message that blocks everything behind it

A backlog nobody watches

Expecting order across partitions

Do it on your own machine

Three questions before you move on

Two questions from earlier lessons

Answer in your head, then check

Related deep dives on this site

Sources