System design, step by step · Lesson 4 of 7 · 14 min

Consistency in Distributed Systems Explained: ACID, CAP, Sagas

Consistency in distributed systems, step by step: ACID and isolation on one node, CAP and PACELC, two-phase commit versus sagas, and leases with fencing.

By Matthews WongReviewed on 13 October 2026

Key takeaway

Consistency in a distributed system is the promise about what a read returns and which writes survive once data lives in more than one place. On one node, ACID transactions and isolation levels keep it. Across replicas, CAP and PACELC trade it against availability and latency. Across services, two-phase commit keeps it by blocking, and a saga by compensating.

You will be able to

Tell the C in ACID apart from the C in CAP, and choose a fix for a read-then-write race.
Predict what two-phase commit does when its coordinator dies, and what a saga does when a step fails.
Explain why a lease needs a fencing token before it can protect a write.

Watch it work

Step through the diagram

Press Play, or step with the arrows. Every step is written out below the diagram.

Two-phase commit blocks, a saga compensates

One checkout writes stock and payment twice over: with two-phase commit the coordinator dies after both databases vote yes, so their locks stay and the next order waits; with a saga each step commits at once, the declined payment triggers a compensation, and an order in between sees stock that was later put back.

One checkout writes to two places, stock and payment. Top: two-phase commit, run by a coordinator. Bottom: a saga, run by an orchestrator. Same order, two protocols.

Keys: ← → step · Home / End · K play or pause

One checkout writes to two places, stock and payment. Top: two-phase commit, run by a coordinator. Bottom: a saga, run by an orchestrator. Same order, two protocols.
Phase one: the coordinator asks both databases to prepare. Each makes its change durable, keeps its row locks and votes yes, then waits for the decision.
BEGIN; UPDATE stock SET qty = qty - 6 WHERE item = 7; PREPARE TRANSACTION 'checkout-42';
The coordinator crashes before it sends the decision. Both databases voted yes, but neither may commit or abort alone: the other might have been told the opposite.
SELECT gid, prepared FROM pg_prepared_xacts;Predict the next step The coordinator died after the votes. What happens to the next order? (It waits behind the row lock · The lock times out, so it succeeds · It reads the old stock and passes)
So the locks stay. The next order for the same item queues behind the stock row until the coordinator returns. Nothing times a prepared transaction out.
UPDATE stock SET qty = qty - 5 WHERE item = 7; -- blocks on the prepared transaction's row lock
The saga commits each step at once. Stock reserves 6 units and commits, leaving 4. Then the payment is declined. No lock was held across either call.
INSERT INTO reservations (id, item, qty, status) VALUES (42, 7, 6, 'held');
Meanwhile another order wants 5. Nothing is locked, so it is answered at once, but it sees only 4 left and is refused. A saga gives no isolation.
Predict the next step Payment was declined. How does the saga undo the stock reservation? (The database rolls back step one · A new transaction puts the units back · It waits for a coordinator decision)
The orchestrator runs the compensation: a new local transaction that puts the 6 units back. Stock reads 10 again, but the refused order saw a state that never became real.
-- a new local transaction; running it twice changes nothing UPDATE reservations SET status = 'released' WHERE id = 42 AND status = 'held';

What does consistency mean inside a single database?

On one node, the transaction is the unit of consistency, and ACID names its four guarantees. Atomic: all of it happens or none of it does. Consistent: it moves the database from one valid state to another, where valid means the rules you declared, such as CHECK, UNIQUE and foreign keys. Isolated: concurrent transactions do not see each other's half-finished work. Durable: once committed, it survives a crash. The C is the odd one out, because the database enforces only the invariants you actually wrote down.

Read more (1)

Isolation is where most bugs hide. PostgreSQL's default level, Read Committed, gives each statement a snapshot taken when that statement starts, so two statements in one transaction can see different data. Read a balance, subtract in application code and write the result back, and two concurrent requests silently lose one update. Do the arithmetic inside one UPDATE, or lock the row first with SELECT FOR UPDATE. Repeatable Read and Serializable prevent more anomalies, but the application must then retry whenever PostgreSQL reports a serialization failure, SQLSTATE 40001.

Why do CAP and PACELC force a trade once data is replicated?

The C in CAP is a different promise: every read receives the most recent write or an error, as if there were one copy. Availability means every request to a working node gets a non-error answer. When a network partition cuts the replicas apart, you cannot have both. A CP system refuses some requests on the minority side rather than return stale data; an AP system keeps answering on both sides and reconciles later. Partitions are not optional on a real network, so pick two out of three is the wrong summary: the choice only bites during a partition.

Read more (1)

PACELC adds the part you live with every day: else, when the network is healthy, you trade latency against consistency, because waiting for a replica to confirm costs at least one round trip per write. Systems that choose speed are eventually consistent: in Werner Vogels' words, if no new updates are made, eventually all accesses return the last updated value. That promises where replicas end up, not what a reader sees on the way. Reads are cheap to be wrong about; decisions are not. A balance, the last unit of stock at checkout and a unique username need one strongly consistent check at the moment of decision.

How do two-phase commit and sagas keep several services in step?

Two-phase commit puts a coordinator in charge. In the prepare phase every participant makes its change durable, keeps its locks and votes; in the commit phase the coordinator commits only if every vote was yes. The weakness is blocking: a participant that voted yes must wait for the decision, and if the coordinator never returns, it holds its locks indefinitely. PostgreSQL supports the participant side with PREPARE TRANSACTION, whose state is stored on disk and can be finished from any session, but its documentation says the command is meant for external transaction managers, not applications, and warns that a prepared transaction keeps holding its locks and gets in the way of VACUUM.

Read more (1)

A saga replaces the one big transaction with a sequence of local ones, each committed immediately. When a step fails, the saga runs compensating transactions that undo the earlier steps, either directed by an orchestrator or triggered by events in a choreography. A compensation is a new forward transaction, such as a refund or a release, so it must be safe to run twice and must be retried until it succeeds. The price is isolation: other transactions see the intermediate state, so teams add countermeasures such as a PENDING status on the row. Before choosing either, ask whether one database could hold all the data, because one local transaction beats both.

Why does a lease need a fencing token?

Some work must run on exactly one node: a nightly job, the primary database, the process that hands out assignments. A lease elects that node: a lock with an expiry that the holder keeps renewing, so a crashed holder eventually loses it. The trap is the gap between checking the lease and doing the work. Martin Kleppmann's example is a client that stalls in a long garbage-collection pause, wakes after its lease expired and writes as if nothing happened. Checking the expiry just before writing does not help, because the pause can land between the check and the write.

Read more (1)

The fix is a fencing token: a number that goes up every time the lock is granted. The holder sends it with every write, and the protected resource rejects any write whose token is lower than one it has already seen. The check has to live in the resource, in the same statement as the write, not in the leader's own code. Kleppmann draws the line clearly: a lock used only to avoid duplicate work can be loose, but a lock used for correctness needs a token. Raft's term numbers do the same job inside a consensus cluster, and its post has its own step-through diagram.

How this site does it

The real config behind this lesson

Excerpts from the repository that builds and deploys this website, quoted as they are.

.github/workflows/deploy.yml:189-196

    # The one group that must NEVER cancel in progress. This step SSHes to the
    # VPS and runs `docker compose pull && up -d`; interrupting it can leave the
    # box between images. Concurrent deploys queue and run one at a time, which
    # is the original intent — it is only the build chain above that should be
    # throwing away superseded work.
    concurrency:
      group: deploy-vps
      cancel-in-progress: false

This site has no distributed database, but its deploy has the shared-state problem in miniature: several pushes want to change one server. The concurrency group is a lock that GitHub holds, so at most one deploy runs at a time, and cancel-in-progress: false means a running deploy is never interrupted, because pull and up -d are not one atomic step. GitHub keeps at most one run pending and cancels an older pending one when a newer arrives. That last-writer-wins rule is safe here only because every deploy writes the whole desired state, never an increment.

.github/workflows/deploy.yml:20-22

# So: builds cancel, deploys never do. A cancelled build also cancels its own
# deploy before it can start (deploy declares `needs: build-and-push`), so
# there is no path to a half-applied deploy.

The same idea a saga uses: put the work that may fail or be abandoned before the step that cannot be safely interrupted. Verification and the image build can be cancelled at any moment, because nothing outside the runner has changed yet. The step that touches the server only starts once everything before it has succeeded, and from then on it is never cancelled.

What breaks

Mistakes that cost real time

Read, compute, write at Read Committed
Selecting a value, changing it in application code and writing the result loses updates under concurrency, with no error. Move the arithmetic into the UPDATE (balance = balance - 30, with a CHECK constraint behind it) or lock the row with SELECT FOR UPDATE.
A prepared transaction nobody finishes
A prepared transaction survives restarts on purpose, so nothing expires it. Its row locks block every writer and it holds back VACUUM. If you enable max_prepared_transactions, something must own pg_prepared_xacts and alert when a row there grows old.
A compensation that is not safe to repeat
Compensations are retried until they succeed, often during the same outage that failed the step. A release written as qty = qty + 6 adds the units twice on a retry. Make it a state change that matches nothing the second time, such as status = 'released' WHERE status = 'held'.
Last-write-wins on a number that must add up
Last-write-wins converges, so every replica agrees, while it silently throws one of two concurrent writes away. On stock of 10 with 3 sold on one replica and 4 on another, it keeps one side and reports 7 or 6 instead of 3. Use it only where losing one of two edits is genuinely fine.
A lease checked by the leader, not by the resource
If only the leader checks its lease, a paused leader that wakes up late still writes, and for a moment two nodes act as leader. Pass a fencing token with every write and let the storage reject the stale one.

Two-phase commit or saga: what each one gives and costs
	Two-phase commit	Saga
Atomicity	All commit or none do, decided by the coordinator	Each step commits alone; the whole is undone by compensations
Isolation	Locks hide the half-done state until the decision	None: others see the intermediate state
When the brain fails	Participants wait, holding locks	Nothing is locked; the saga resumes or compensates
Every participant must	Support prepare, like PostgreSQL or an XA resource	Have a local transaction and a safe undo
Fits best	A few databases you run yourself, short transactions	Flows across services or vendors, such as a payment gateway

Try it yourself

Do it on your own machine

You need Docker and nothing else. Start PostgreSQL with prepared transactions enabled, play the participant whose coordinator never came back, and watch a second writer get stuck behind it.

Start PostgreSQL 18 with prepared transactions switched on (the default, 0, turns them off), then create a stock row.

docker run --rm -d --name pg2pc -e POSTGRES_PASSWORD=lab \
  postgres:18 -c max_prepared_transactions=10

docker exec pg2pc psql -U postgres \
  -c "CREATE TABLE stock (item int PRIMARY KEY, qty int NOT NULL CHECK (qty >= 0))" \
  -c "INSERT INTO stock VALUES (7, 10)"

Be the participant: change the row, vote yes with PREPARE TRANSACTION, and let the session end without a decision.

printf "BEGIN;\nUPDATE stock SET qty = qty - 6 WHERE item = 7;\nPREPARE TRANSACTION 'checkout-42';\n" \
  | docker exec -i pg2pc psql -U postgres

3
Be the next order. With a 3-second lock_timeout the UPDATE gives up with "canceling statement due to lock timeout" instead of waiting forever.
```
docker exec pg2pc psql -U postgres \
  -c "SET lock_timeout = '3s'" \
  -c "UPDATE stock SET qty = qty - 5 WHERE item = 7"
```

Restart the server: the prepared transaction is still listed. Finish it from a new session, and the blocked UPDATE now succeeds.

docker restart pg2pc
docker exec pg2pc psql -U postgres -c "SELECT gid, prepared FROM pg_prepared_xacts"
docker exec pg2pc psql -U postgres -c "ROLLBACK PREPARED 'checkout-42'"
docker exec pg2pc psql -U postgres -c "UPDATE stock SET qty = qty - 5 WHERE item = 7 RETURNING qty"
docker rm -f pg2pc

Quick quiz

Three questions before you move on

Pick an answer to see why it is right or wrong. Nothing is saved or graded, so guess freely.

1.Two cashiers sell from the same balance at the same second. Your code runs SELECT balance, subtracts in TypeScript, then UPDATEs with the result, at PostgreSQL's default isolation level. What happens?
2.In two-phase commit, the coordinator crashes after both databases voted yes and before it sent the decision. What do the databases do?
3.A leader holding a 15-second lease freezes for 20 seconds, then wakes up and writes. What reliably stops that stale write?

Think it through

Answer in your head, then check

1.Does the C in ACID mean the same as the C in CAP?Show the answer
Answer:No. ACID's consistency means a transaction keeps the rules you declared, such as CHECK and UNIQUE constraints. CAP's consistency means every read sees the most recent write or an error, as if there were one copy.
2.Why can't a participant that voted yes simply roll back when the coordinator goes quiet?Show the answer
Answer:Because another participant may already have been told to commit. Rolling back alone could leave half the transaction committed, which is exactly what two-phase commit exists to prevent. So it waits, and its locks wait with it.
3.A saga reserved stock, then the payment failed. Why is the fix called a compensation and not a rollback?Show the answer
Answer:The reservation already committed and others may have seen it, so there is nothing left to roll back. The saga runs a new local transaction that releases the stock, and it must be safe to repeat.
4.This site's deploy lets a newer pending run cancel an older one. Why is that safe here?Show the answer
Answer:Each deploy writes the whole desired state, the newest image, rather than a change on top of the last one. Skipping an intermediate full state loses nothing, which is why last-writer-wins is fine for it and wrong for a counter.

Read the module

Every post in this module, in order

The lesson above is the map. Each post goes deep on one part of it, with the full worked examples.

Sources

Primary sources checked on the research date. Follow them for the current version.

Previous lessonHow to Scale a Database: Indexes, Replicas and Sharding Next lessonReliability Patterns Explained: Retries to Circuit Breakers

Step through the diagram

Two-phase commit blocks, a saga compensates

What does consistency mean inside a single database?

Why do CAP and PACELC force a trade once data is replicated?

How do two-phase commit and sagas keep several services in step?

Why does a lease need a fencing token?

The real config behind this lesson

Mistakes that cost real time

Read, compute, write at Read Committed

A prepared transaction nobody finishes

A compensation that is not safe to repeat

Last-write-wins on a number that must add up

A lease checked by the leader, not by the resource

Do it on your own machine

Three questions before you move on

Answer in your head, then check

Every post in this module, in order

Sources