DevOps
Docker Swarm Load Testing: Why More Replicas Didn't Fix Throughput
July 202612 min read
Photo by Robert Scoble via Openverse (CC BY 2.0)
Doubling replicas splits incoming HTTP concurrency across two instances, which halves per-replica queue depth and cuts p95 tail latency (8.40s to 4.50s in this case). Throughput is a different metric bounded by the shared PostgreSQL database, which does not gain capacity just because a second application replica was added. Throughput only rose 9.3% because the database write path, not the HTTP layer, was the real constraint.
Sample pg_stat_activity joined against pg_locks while load is running, filtering for sessions where granted is false. If many sessions are consistently blocked on a tuple lock against the same table and row, that row is serializing every write that touches it. In this case, six to eight sessions were repeatedly blocked on one row in a tenant_config table that every request updated.
No. Row-level locking in PostgreSQL only allows one writer to hold an exclusive lock on a given row at a time, regardless of how many connections, pool size, or pooling proxies sit in front of it. In this research, a HikariCP pool sweep, a fillfactor change, and PgBouncer transaction pooling all failed to lift throughput above 2.27 iterations per second, and PgBouncer actually made it 13% worse due to per-query overhead.
Yes, when the bottleneck is a shared resource that all replicas contend for, such as a single database row, a shared cache key, or a shared external API rate limit. Horizontal scaling helps when the constraint is per-instance CPU or connection concurrency, but it does nothing, or can even hurt, when the constraint sits downstream at a shared, serialized resource.
In this research, three changes on a single untuned replica raised throughput by 32% and cut p95 latency by 57%: setting synchronous_commit to off to skip the per-commit fsync wait, increasing shared_buffers and wal_buffers to reduce disk I/O and smooth WAL flushes, and widening the HikariCP connection pool from 5 to 10 per replica using the standard cores times two plus spindle count sizing rule.
Photo by Robert Scoble via Openverse (CC BY 2.0)
Key Takeaway
In a Docker Swarm load test on a Java Spring Boot logistics platform, doubling replicas cut p95 latency by half but only raised throughput 9 percent, because every request updated the same PostgreSQL row. Tuning PostgreSQL and HikariCP on a single replica beat two tuned replicas, proving horizontal scaling cannot fix a shared-resource row lock.
During my undergraduate thesis at Swiss German University, I ran SRE-style performance research at Commsult Indonesia on Ontego Traces, a Java Spring Boot logistics SaaS running on Docker Swarm for the German market. The platform had never been load tested. Every service ran as a single replica, nobody knew its breaking point, and the only signal anyone had was a log stream with no timing data attached. I built a k6-based load testing setup around the platform's busiest write path, the Create Tour endpoint, expecting the obvious fix for slow performance to be adding more replicas. The results argued otherwise, and chasing down why took me from a Docker Swarm scale command all the way to a single locked row in a PostgreSQL table.
Ontego Traces is a six-service Java Spring Boot platform built by Commsult AG. Before this project, its only observability was log aggregation, which recorded errors but never response times or throughput. Every service, including the Tour Data Provider that ingests every new tour, ran a single container replica in Docker Swarm. A prior month-long trace of the health endpoint showed visible downtime and response spikes above 18 seconds during earlier uncontrolled testing, but nobody could say what load the platform could safely absorb, or which layer would give out first. That absence of a baseline was the actual problem I was there to fix, not any specific bug.
I picked the Create Tour endpoint on the Tour Data Provider as the single highest-information target. Every call validates an OAuth2 token against Keycloak, deserialises a nested payload of stops, orders, and order positions, issues 87 PostgreSQL INSERT statements in one transaction, and publishes a message to an Artemis queue on success. One endpoint exercises authentication, deep persistence, and asynchronous messaging in a single request path, so a bottleneck anywhere in the stack would show up here first.
I captured a real production Create Tour payload and stripped every PII field, customer numbers, names, emails, addresses, order identifiers, replacing them with null or synthetic values while keeping the same nine-stop, nested-order structure. That mattered because a flattened or fake payload would have skipped the exact serialization and persistence code paths I needed to stress. Every comparative experiment ran the identical k6 script for 10 virtual users over 120 seconds; a pilot phase showed indistinguishable results below 10 VUs and load-generator CPU climbing past 70 percent above 20 VUs, so 10 VUs became the fixed profile for every run that followed.
If you are load testing a platform that touches customer data, anonymize a real captured payload instead of hand-writing a synthetic one. Keeping the same nesting depth and field structure is what makes the load test exercise the real serialization and persistence code, not a simplified stand-in.
My first hypothesis was straightforward: doubling replicas under 10 virtual users should meaningfully cut tail latency. It did, dramatically. p95 response time fell from 8.40 seconds at one replica to 4.50 seconds at two, a 1.87x improvement, and the maximum response time roughly halved as well. But throughput, the number that determines how many tours the platform can actually process per second, moved from 1.72 to 1.88 iterations per second: a 9.3 percent gain for double the compute. The latency win was real: Docker Swarm's round-robin load balancer was splitting the queue of 10 concurrent virtual users across two instances instead of one, which halves per-replica concurrency and shortens the wait for the slowest requests. But something downstream of the application layer was capping how many tours could complete per second, regardless of how many replicas handled the HTTP layer.
Before touching the database, I checked the obvious alternative suspect: the message queue that the endpoint publishes to on every successful tour. I polled queue statistics every 2 seconds during both runs. The publish rate was 57 messages versus 54 over the same 120 seconds, essentially unchanged, and the queue drained to zero within about 10 seconds in both cases with one active consumer. A saturated consumer would show a growing backlog; there was none. The broker was healthy. The bottleneck sat somewhere between the application and the database.
-- 1. PostgreSQL runtime tuning applied via ALTER SYSTEM (no restart needed
-- for synchronous_commit; shared_buffers/wal_buffers need a restart)
ALTER SYSTEM SET synchronous_commit = off; -- skip per-COMMIT fsync wait
ALTER SYSTEM SET wal_buffers = '64MB'; -- was 4MB default
ALTER SYSTEM SET shared_buffers = '1GB'; -- was 128MB, ~25% of host RAM
SELECT pg_reload_conf();
# 2. HikariCP pool widened from 5 to 10 per replica
# (core_count x 2) + effective_spindle_count = (4 x 2) + 1 = 9, rounded to 10
hikari:
maxLifeTime: 600000
minimum-idle: 10
maximum-pool-size: 10
# Result on a SINGLE replica: 1.72 -> 2.27 iter/s (+32%), p95 8.40s -> 3.65s (-57%)
# That single tuned replica beat every multi-replica config tested, including
# 2 tuned replicas (2.00 iter/s, p95 5.43s) -- which is where the story gets interesting.
Rather than add more replicas, I tuned the shared PostgreSQL instance and its connection pool. Three changes, shown below, were applied without touching application code: turning synchronous_commit off to skip the per-commit fsync wait, a larger shared_buffers and wal_buffers to keep more data in memory and smooth WAL flushes, and a HikariCP pool widened from 5 to 10 connections per replica, sized by the standard cores times two plus spindle count rule for a 4-core host. On a single, untouched replica, throughput rose from 1.72 to 2.27 iterations per second, a 32 percent gain, while p95 latency fell from 8.40 to 3.65 seconds, a 57 percent reduction. That single tuned replica beat every multi-replica configuration I tested, including two replicas running the exact same tuning, which only reached 2.00 iterations per second with a p95 of 5.43 seconds. Fixing the actual constraint outperformed doubling the compute sitting in front of it.
Two tuned replicas were slower than one. Before tuning, the bottleneck was WAL fsync latency and a starved 5-connection pool, so scaling to two replicas doubled available connections and helped a little. After tuning removed those constraints, two replicas doubled the number of connections competing for something tuning could not fix: a single row lock. Don't assume the fix that worked at one layer will keep working once you have removed the constraint underneath it.
With the broker cleared and PostgreSQL tuned, I sampled pg_stat_activity joined against pg_locks every 3 seconds during a live two-replica run to see exactly which sessions were blocked and on what. Every sample looked the same: six to eight sessions stuck on a tuple lock, all running the identical UPDATE against a tenant_config table. The staging environment had 40 rows in that table, one per tenant, and every single load test request used the same tenant key, so every concurrent Create Tour transaction was fighting for an exclusive lock on one row. The table already had a 98.9 percent HOT update ratio, meaning PostgreSQL was already reusing heap space about as efficiently as it could; the cost was not storage, it was the lock itself.
-- Sampled every 3s during a 2-replica run to see who is blocked and on what
SELECT
blocked.pid,
blocked.wait_event,
blocked_locks.locktype,
blocked_locks.relation::regclass AS locked_table,
blocked.query
FROM pg_catalog.pg_locks blocked_locks
JOIN pg_catalog.pg_stat_activity blocked
ON blocked.pid = blocked_locks.pid
WHERE NOT blocked_locks.granted
LIMIT 10;
-- Every sample looked the same: 6-8 sessions stuck on a tuple lock,
-- all running the identical UPDATE:
--
-- pid | wait_event | locktype | locked_table | query
-- -----+------------+----------+----------------+----------------------------
-- 203 | tuple | tuple | tenant_config | update tenant_config set
-- 350 | tuple | tuple | tenant_config | modified_at=$1, tenant=$2,
-- 349 | tuple | tuple | tenant_config | tenant_info=$3, tenant_key=$4
-- 199 | tuple | tuple | tenant_config | where id=$5
-- 201 | tuple | tuple | tenant_config |
-- 198 | tuple | tuple | tenant_config |
-- 347 | tuple | tuple | tenant_config |
--
-- tenant_config had 40 rows (one per tenant). Every load-test request used
-- tenantKey = "BEHNGT" -- one row, serialised by Postgres row-level locking.
Once I knew the row was the constraint, I tried every infrastructure lever available. A pool size sweep across two tuned replicas found pool size 4 per replica, 8 total connections, as the least-bad configuration at 2.18 iterations per second and a p95 of 4.31 seconds, still 4 percent below the single-replica tuned result and 18 percent worse on p95. Setting the table's fillfactor to 50 to give HOT updates more room did nothing measurable. Routing through PgBouncer in transaction-pooling mode actually regressed throughput by about 13 percent, because its per-query overhead compounds across the 87 INSERT statements in every transaction. Nothing crossed 2.27 iterations per second. That number is not a PostgreSQL setting; it is the maximum rate at which transactions can acquire and release a single row lock, one at a time, no matter how the surrounding infrastructure is arranged.
The single-replica tuned configuration, not the two-replica configuration, became the production recommendation: better throughput, better latency, and half the compute cost. Sometimes the fastest fix is not adding capacity, it is reading the lock table.
Adding replicas relieves whatever bottleneck sits at the HTTP concurrency layer, and that is a real, visible win: tail latency nearly halved in this project. But replicas share the same database, and when the actual constraint is a single row that every request needs to update, more replicas just means more processes queuing for the same lock. No PostgreSQL parameter, connection pool size, storage setting, or pooling proxy crossed that ceiling, because none of them touch what row-level locking enforces: only one writer holds an exclusive lock on a given row at a time. The only real fix is application-level: restructuring the write so concurrent requests stop contending for the same row, caching it in-process, splitting it by tenant, or dropping the write from that transaction entirely. Diagnosing that took going past dashboards, past connection pool metrics, down to a live query against pg_locks. That is usually where the real bottleneck in a write-heavy system is actually hiding.