Skip to main content
DevOps from zero · Lesson 8 of 8 · 12 min

Monitoring Docker Containers: Uptime, Rollback and Backups

Learn what to watch on a Docker server and why: health checks, uptime checks and metrics. Then roll back a broken release by its tag and test a backup.

By Matthews WongReviewed on

Key takeaway

Monitoring Docker containers means watching them from three sides: the container's own health check, an uptime check from outside, and metrics over time. Monitoring only tells you something broke. Recovery is a rollback, which means starting the previous image again by its immutable tag. Backups cover what a rollback cannot bring back: data.

You will be able to

  • Tell a health check, an uptime check and a metric apart, and say what each one catches.
  • Roll a broken release back to the previous image tag, and explain why detecting a failure is not recovering from it.
  • Decide what on one server needs a backup, and prove it with a restore.
Watch it work

Step through the diagram

Press Play, or step with the arrows. Every step is written out below the diagram.

A bad release: who notices, and how you undo it

v2 is deployed, its health check fails while it keeps serving visitors, an outside uptime check alerts you, and starting the previous image tag again brings the service back.

Step 1 of 7. Two tagged images, one server and two watchers. Docker's health check runs inside the web container; Uptime Kuma checks the same app from outside, the way visitors reach it.

Keys: ← → step · Home / End · K play or pause

  1. Two tagged images, one server and two watchers. Docker's health check runs inside the web container; Uptime Kuma checks the same app from outside, the way visitors reach it.

  2. The deploy points IMAGE at the v2 tag and runs Compose, which replaces the v1 container with a v2 one. The build passed, so everything looks green.

    IMAGE=devops-lab:v2 \ docker compose up -d --wait

  3. Inside the container, /health now answers 500. After three failed checks, Docker marks v2 unhealthy and --wait exits 1. The deploy noticed first, but it rolls nothing back.

    interval: 5s retries: 3Predict the next step v2 is now unhealthy. What do visitors get? (An error page: Docker stopped v2 · Pages from v1, switched automatically · Pages from v2, as before)

  4. Visitors still get pages from v2. Unhealthy is a label, not an off switch: restart policies react to the process exiting, and this process is still running.

    curl http://127.0.0.1:8080/ hello from v2

  5. Uptime Kuma calls /health from outside, gets the 500 and alerts you. Without --wait, as in this site's deploy, this is the first warning a human gets.

    docker compose up -d --remove-orphansPredict the next step You got the alert. What is the fastest way back to v1? (Point IMAGE at v1 and rerun Compose · Rebuild v1 from source, then deploy)

  6. Roll back: point IMAGE at the previous tag and run Compose again. That tag names one exact build, still on disk, so nothing has to be rebuilt.

    IMAGE=devops-lab:v1 \ docker compose up -d --wait

  7. v1 passes its health check, --wait exits 0 and the probe reports up again. Now fix v2 calmly: the rollback bought time, it did not fix the bug.

    curl http://127.0.0.1:8080/ hello from v1

What is the difference between a health check, an uptime check and metrics?

A health check runs inside the container. Docker executes a command on a schedule, here a request to /health every few seconds, and marks the container healthy or unhealthy. It is cheap and fast, but it only proves that the process answers on its own loopback. It cannot see DNS, TLS, the reverse proxy or the network between your server and your visitors.

Read more (3)

An uptime check calls your public URL from outside, the way a visitor would, and alerts someone when it stops answering. Uptime Kuma is a self-hosted tool for exactly this: HTTP, TCP, ping and keyword monitors, with notifications to Telegram, Slack, email and many other channels. Run it somewhere other than the server it watches, or it goes down together with the thing it is meant to report.

Metrics are numbers over time: memory, CPU, request counts. Prometheus collects them by pulling over HTTP from exporters on a schedule, node exporter for the host and cAdvisor for containers, and Grafana draws them as dashboards. Metrics answer a different question from the other two: not whether the app is up, but whether it is getting worse, such as memory creeping towards the container's limit before the kernel kills it.

This site's own Prometheus, Grafana and Uptime Kuma configuration lives on the server, not in the repository, so it is not public and this lesson does not quote it. Every tool here runs on your laptop from its official image instead, and the setup guides linked at the end go through each one step by step.

Why does a failed health check not roll anything back?

Because detecting a problem and recovering from it are different jobs, and Docker only does the first. It takes retries consecutive failures for Docker to mark a container unhealthy, and that is a label. The process keeps running and keeps answering requests, because restart policies react to the process exiting, not to its health.

Read more (2)

docker compose up --wait makes the deploy notice. It waits for services to be running or healthy, and exits with an error when one turns unhealthy. In the drill for this lesson, a v2 whose /health answered 500 made --wait exit 1 after about 19 seconds, with checks 5 seconds apart. curl still got hello from v2. The release was broken and live at the same time.

This site's deploy runs docker compose up -d without --wait, so a release that fails its health check still finishes the workflow green. Adding --wait turns a silent failure into a red run. It still would not roll back: that step is yours, or a script's.

How do immutable image tags make a rollback fast?

A rollback means running the previous known-good version again. That is only quick when the previous version has a name that points at exactly one build. latest moves with every push, so after a bad deploy it points at the bad build. An immutable tag is one you never reuse: tag every build with its commit SHA and that tag always means that build.

Read more (2)

This site's pipeline pushes every image twice, as latest and as the commit SHA, and the deploy writes the SHA tag into the server's .env as IMAGE. So the live version is always one exact build, and the one before it has a name too. The cleanup after each deploy keeps images created in the last 72 hours, so a recent rollback starts from a copy already on disk.

There are three ways back, from fastest to most traceable. On the server, set IMAGE to the previous tag and run docker compose up -d --wait; the next deploy rewrites .env, so treat this as a stopgap. In GitHub, re-run the last good workflow run, which uses the same commit SHA and is allowed up to 30 days after the original run. Or git revert the bad commit and push, which goes through the whole pipeline and leaves an audit trail.

What should you back up on one server, and how do you test a restore?

Start from where the state lives, not from the disk. For this site the answer is almost nowhere on the server: the code is in git, every image is in the registry, and the configuration is in GitHub secrets and variables, so a fresh server can be rebuilt from those. The one volume the app mounts holds a regenerable image cache, not data.

Read more (2)

Other services are different. A monitoring tool keeps its settings and history in a volume, and a database keeps everything there. Those volumes are your backup list, and they are why this site's cleanup never prunes volumes: pruning them would destroy their data. Keep at least one copy off the server, because a backup on the same disk is lost with that disk.

A backup you have never restored is only a hope. Docker's own recipe mounts the volume into a throwaway container and writes a tar archive to a host folder; restoring is the same in reverse, into a new volume. Then start a second copy of the service on the restored volume and check that your data is there. That last step is the actual test.

How this site does it

The real config behind this lesson

Excerpts from the repository that builds and deploys this website, quoted as they are.

docker-compose.yml:29-47
    healthcheck:
      test:
        [
          "CMD",
          "wget",
          "--no-verbose",
          "--tries=1",
          "--spider",
          "http://127.0.0.1:3000/robots.txt",
        ]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 20s
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"

The first two instruments live in the runtime file itself. Every 30 seconds Docker runs BusyBox wget against /robots.txt inside the container, three failures in a row mark it unhealthy, and the 20-second start period gives Next.js time to boot. A cheap URL like this proves the Node server answers, not that every page renders. The logging block caps each container at three 10 MB files, so logs cannot fill the disk.

.github/workflows/deploy.yml:120-127
      - name: Build and push
        uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          tags: |
            ${{ steps.meta.outputs.ref }}:latest
            ${{ steps.meta.outputs.ref }}:${{ github.sha }}

Every image is pushed twice, as latest and as the commit SHA. The deploy job writes the SHA tag into the server's .env as IMAGE (line 231), so the running version is always one exact build, and so is the one before it. That second, never-reused name is the whole rollback mechanism.

.github/workflows/deploy.yml:247-259
# Reclaim disk once the new container is serving.
#
# Every deploy pulls a fresh :<sha> tag, and a tagged image is not
# dangling, so the previous `docker image prune -f` never removed
# them — they accumulated on the VPS until the disk filled. `-a`
# drops the superseded tags too, while `until=72h` keeps the last
# few days so a rollback can still start from a local image.
#
# Images backing a running container are never removed, which is
# what keeps the other stacks on this box (<other services>)
# intact. Volumes are deliberately left alone —
# pruning those would destroy their data.
docker image prune -af --filter "until=72h"

The cleanup after every deploy, and the trade-off it makes. Without -a, old SHA-tagged images piled up until the disk filled; with -a alone, the previous release would be deleted too. The until=72h filter removes only images created more than 72 hours ago, so a recent rollback starts from a local copy. A placeholder replaces the names of the other services on the box.

What breaks

Mistakes that cost real time

  • Unhealthy, and still live

    A failed health check changes a status word in docker ps and nothing else. The container keeps serving, and restart policies do not react to health. Pair --wait with a rollback you have rehearsed, or a broken release stays up until someone notices.

  • What I'd change here: the deploy does not wait

    This site's deploy runs docker compose up -d --remove-orphans and moves on, so a release that fails its health check still ends the workflow green. Adding --wait, with --wait-timeout as a ceiling, would turn that run red, which is the signal to roll back.

  • Deploying latest, then having nothing to go back to

    If the server runs whatever latest points at, a bad push overwrites your way back. Deploy the SHA tag instead, as this site's .env does, and the previous release always has an exact name.

  • Cleaning up too hard

    docker image prune -a with no filter removes every image no container uses, including the release you may need tonight, which then has to be pulled from the registry again, possibly behind a login. Pruning volumes is worse: that deletes data. Filter images by age and leave volumes alone.

  • Watching only from inside, or exposing the dashboards

    A container can be healthy while DNS, TLS or the proxy in front of it is broken; only an outside check sees that. And the quick-start commands for Uptime Kuma, Prometheus and Grafana publish their ports on every interface. Bind them to 127.0.0.1 and reach them over an SSH tunnel.

Three ways to watch one server
Runs whereInside the container, run by DockerOutside, the way visitors arriveOn the server, scraped on a schedule
AnswersDoes the process answer?Can visitors reach it?Is it getting worse?
CatchesA crashed or stuck appDNS, TLS, proxy and network failures tooSlow trends in memory, CPU and disk
MissesEverything outside the containerWhy it failedNothing alerts until you add alert rules
Tool in this lessonCompose healthcheckUptime KumaPrometheus and Grafana
Try it yourself

Do it on your own machine

The rollback drill. You need Docker and the lab folder from the earlier lessons: the Node server with a /health route, its multi-stage Dockerfile and compose.yaml. You will ship a broken release on purpose, watch it stay live, and roll it back.

  1. Check compose.yaml: the image tag comes from IMAGE, the port is published on loopback only, and the health check calls /health every 5 seconds.

    services:
  web:
    image: ${IMAGE:-devops-lab:v1}
    restart: unless-stopped
    environment:
      APP_VERSION: ${APP_VERSION:-v1}
    ports:
      - "127.0.0.1:8080:3000"
    mem_limit: 128m
    cpus: "0.5"
    healthcheck:
      test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://127.0.0.1:3000/health"]
      interval: 5s
      timeout: 3s
      retries: 3
      start_period: 5s
    logging:
      driver: json-file
      options:
        max-size: "1m"
        max-file: "3"

  2. Build v1, start it and wait until Docker reports it healthy.

    docker build -t devops-lab:v1 .
docker compose up -d --wait
curl http://127.0.0.1:8080/    # hello from v1

  3. Ship a bug: in src/server.ts, make the /health branch answer 500 instead of 200. Build it as v2 and deploy it with --wait. After about 20 seconds Compose reports the container unhealthy and exits 1, yet curl still gets an answer from v2.

    # src/server.ts, inside the /health branch
res.writeHead(500, { "content-type": "text/plain" });

docker build -t devops-lab:v2 .
IMAGE=devops-lab:v2 APP_VERSION=v2 docker compose up -d --wait; echo "exit=$?"
curl http://127.0.0.1:8080/    # hello from v2: broken and live
docker compose ps              # STATUS ends in (unhealthy)

  4. Roll back by starting the previous tag. --wait exits 0 once v1 is healthy, which took 8 seconds in the test run, and the log names the version that is running.

    IMAGE=devops-lab:v1 APP_VERSION=v1 docker compose up -d --wait; echo "exit=$?"
curl http://127.0.0.1:8080/    # hello from v1
docker compose logs --tail 20 web

  5. Optional, the outside view: add Uptime Kuma to compose.yaml, published on loopback only. Open http://127.0.0.1:3001, finish the first-run setup and add an HTTP monitor for http://web:3000/health. Repeat steps 3 and 4 and watch it report DOWN, then UP. It checks on its own interval, so it can lag behind --wait: that is who noticed first.

    # under services:
  uptime-kuma:
    image: louislam/uptime-kuma:2
    restart: unless-stopped
    ports:
      - "127.0.0.1:3001:3001"
    volumes:
      - uptime-kuma:/app/data

# at the end of the file, top level:
volumes:
  uptime-kuma:

  6. Optional, the restore drill: stop Uptime Kuma, archive its volume with a throwaway container, restore the archive into a new volume and start a second copy on it at http://127.0.0.1:3002. If your monitor is listed there, the backup works.

    docker compose stop uptime-kuma
docker run --rm --volumes-from "$(docker compose ps -aq uptime-kuma)" -v "$PWD":/backup alpine tar czf /backup/kuma-backup.tgz -C /app/data .
docker compose start uptime-kuma

docker volume create kuma-restore
docker run --rm -v kuma-restore:/data -v "$PWD":/backup alpine tar xzf /backup/kuma-backup.tgz -C /data
docker run -d --name kuma-check -p 127.0.0.1:3002:3001 -v kuma-restore:/app/data louislam/uptime-kuma:2

# clean up afterwards
docker rm -f kuma-check && docker volume rm kuma-restore
Quick quiz

Three questions before you move on

Pick an answer to see why it is right or wrong. Nothing is saved or graded, so guess freely.

  1. 1.Your server runs whatever :latest points at. A bad push goes out, and you want the previous release back. What is the problem?
  2. 2.To free disk, the deploy now runs docker image prune -a with no filter. An hour later the new release breaks. What has the cleanup cost you?
  3. 3.Uptime Kuma runs on the same VPS as the app it watches. The whole VPS goes down. Who alerts you?
Think it through

Answer in your head, then check

  1. 1.v2 builds fine, but /health returns 500. What does docker compose up -d --wait do?Show the answer

    Answer:It exits 1 and reports the container unhealthy, but v2 keeps serving until you roll back. Compose documents --wait as waiting for services to be running or healthy; it never undoes the deploy.

  2. 2.Your container reports healthy, but visitors get an error page. Which check catches that?Show the answer

    Answer:An outside uptime check. The health check runs inside the container, so it cannot see DNS, TLS or the reverse proxy in front of it.

  3. 3.How do you get yesterday's release back fastest on this site?Show the answer

    Answer:Set IMAGE in the server's .env to yesterday's SHA tag and run docker compose up -d --wait. The cleanup keeps images created in the last 72 hours, so it is still on disk. The next deploy rewrites .env, so follow up with a git revert.

  4. 4.What on this server needs a backup for the portfolio itself?Show the answer

    Answer:Nothing unique. The code is in git, the images are in the registry, the configuration is in GitHub, and the cache volume can be regenerated. Back up the volumes of services that hold data, such as monitoring history, and test the restore.

Go deeper

Related deep dives on this site

Sources

Primary sources checked on the research date. Follow them for the current version.

Previous lessonNginx Reverse Proxy With Docker Compose, Behind Cloudflare