Docker Healthcheck Tutorial: Health States and Restart Policies

• Unhealthy for hours, and nothing happened

  A failing check flips a status and raises an event. It does not restart, stop or unroute the container, so a broken app keeps serving errors until someone looks. Make something act on it: --wait at deploy time, a monitor that alerts on the status or on an outside URL, or Swarm mode, which replaces a task whose container fails its checks.

• Connection refused in the health log, yet the app works

  The check calls localhost, BusyBox wget tries ::1 first, and the app listens on 0.0.0.0, which is IPv4 only. The log says can't connect to remote host: Connection refused on every check. Write 127.0.0.1 in the check, as this site does, or make the app listen on IPv6 as well.

• curl: not found, on every check

  A check copied from a tutorial calls curl, and the image does not have it: node:20-alpine ships BusyBox wget and no curl. In a test the log showed /bin/sh: curl: not found with exit code 127, and the container turned unhealthy while the app answered fine. Use a tool the image has, and test the exact command with docker exec first.

• A crash loop that never ends

  always and unless-stopped restart a container that exits at once, forever, with a growing pause up to a minute. docker ps shows Restarting, and RestartCount keeps climbing. Read docker logs for the reason, and give jobs that should give up on-failure with a limit, such as on-failure:3.

1. 1

   Start nginx with a health check that mirrors this site's, with short timings. Run docker ps right away and again 10 seconds later: starting, then healthy.

   docker run -d --name hc-web --restart unless-stopped -p 127.0.0.1:8090:80 \
     --health-cmd "wget --no-verbose --tries=1 --spider http://127.0.0.1/" \
     --health-interval 5s --health-timeout 3s --health-retries 3 --health-start-period 10s \
     nginx:alpine
   docker ps --filter name=hc-web --format '{{.Names}} {{.Status}}'

   Copy
2. 2

   Read the health record: the status, the current failing streak, and each recent check's exit code and output.

   docker inspect --format '{{.State.Health.Status}} streak={{.State.Health.FailingStreak}}' hc-web
   docker inspect --format '{{range .State.Health.Log}}{{.ExitCode}} {{.Output}}{{end}}' hc-web

   Copy
3. 3

   Break it by deleting the page the check asks for. About 15 seconds later the status is unhealthy, yet nginx still answers (403) and RestartCount is 0, despite unless-stopped.

   docker exec hc-web rm /usr/share/nginx/html/index.html
   # about 15 seconds later:
   docker ps --filter name=hc-web --format '{{.Names}} {{.Status}}'
   curl -I http://127.0.0.1:8090/
   docker inspect --format 'restarts={{.RestartCount}}' hc-web

   Copy
4. 4

   Heal it. One passing check is enough to be healthy again. The events list shows each change of status, not every check; if the last healthy line is missing, run it again a few seconds later.

   docker exec hc-web sh -c 'echo ok > /usr/share/nginx/html/index.html'
   # about 10 seconds later:
   docker inspect --format '{{.State.Health.Status}}' hc-web
   docker events --since 5m --until 0s --filter container=hc-web --filter event=health_status

   Copy
5. 5

   Make an empty folder called health-lab and save this as compose.yaml. CHECK_PATH lets you break the check from the command line, and worker waits for api to be healthy.

   services:
     api:
       image: nginx:alpine
       restart: unless-stopped
       ports:
         - "127.0.0.1:8091:80"
       healthcheck:
         test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://127.0.0.1/${CHECK_PATH:-}"]
         interval: 5s
         timeout: 3s
         retries: 3
         start_period: 10s
     worker:
       image: alpine:3
       command: sleep 600
       depends_on:
         api:
           condition: service_healthy

   Copy
6. 6

   In that folder, start it with --wait: exit 0 within seconds. Then start it again with a check that can never pass. After about 20 seconds Compose prints dependency failed to start, exits 1, and worker is Created but never started.

   docker compose up -d --wait; echo "exit=$?"
   docker compose ps
   docker compose down
   CHECK_PATH=missing docker compose up -d --wait; echo "exit=$?"
   docker compose ps -a

   Copy
7. 7

   Crash containers under three policies, each exiting after 2 seconds. About 20 seconds later: no has 0 restarts, on-failure:3 gave up at 3, and always keeps going. Then stop the always one by hand: it stays down.

   docker run -d --name hc-no --restart no alpine:3 sh -c 'sleep 2; exit 1'
   docker run -d --name hc-onf --restart on-failure:3 alpine:3 sh -c 'sleep 2; exit 1'
   docker run -d --name hc-always --restart always alpine:3 sh -c 'sleep 2; exit 0'
   # about 20 seconds later:
   docker inspect --format '{{.Name}} restarts={{.RestartCount}}' hc-no hc-onf hc-always
   docker stop hc-always
   docker ps -a --filter name=hc- --format '{{.Names}} {{.Status}}'

   Copy
8. 8

   Clean up the containers, then, inside health-lab, the Compose project.

   docker rm -f hc-web hc-no hc-onf hc-always
   docker compose down

   Copy

From: 22. Linux File Permissions Explained: ls -l, chmod, chown, umask · 20. Docker Volume vs Bind Mount: Where Container Data Lives

1. 1.What do exit codes 0, 1 and 2 mean to a health check?Show the answer

   Answer:0 is healthy, 1 is unhealthy, and 2 is reserved, so never return it. In a test, other non-zero codes, such as 127 for a missing command, counted as failures too.

2. 2.worker has depends_on: api with condition: service_healthy. An hour after startup, api turns unhealthy. What does Compose do?Show the answer

   Answer:Nothing. Compose checks the condition only while it starts services. In a test, the dependent service kept running after its dependency turned unhealthy.

3. 3.You stopped a container with docker stop, and then the server rebooted. Does it come back under always? Under unless-stopped?Show the answer

   Answer:Under always, yes: a manual stop holds only until the daemon restarts. Under unless-stopped, no: it stays down until you start it yourself. That difference is the reason to pick unless-stopped for a service you sometimes stop for maintenance.

• what a health endpoint should test, with NestJS Terminus
• nginx load balancer health checks and failover
• Docker Swarm, where a failed health check replaces the container