DevOps
kubectl debug: Fix CrashLoopBackOff & Distroless Pods
August 20269 min read
Photo by Bernd Dittrich on Unsplash
Distroless images ship only your application binary, with no shell, package manager, or coreutils. kubectl exec works by launching a program that already exists inside the container, so with no sh or bash present it fails with an executable-not-found error. Use kubectl debug with an ephemeral container instead, which brings its own tooling image.
A crash-looping container only lives for seconds, so there is no stable process to attach to with kubectl exec. Use kubectl debug with --copy-to to clone the pod and override the crashing container's command with a shell, for example kubectl debug myapp -it --copy-to=myapp-debug --container=myapp -- sh. The copy boots and idles so you can investigate; the original pod is untouched.
--target shares the target container's process namespace with your ephemeral debug container. From a tools image like busybox or netshoot you can then see the app's PIDs, read its /proc, and inspect its environment even though the app image has no debugging tools. Process namespace sharing must be supported by the container runtime, and not every platform allows it.
No. Per the Kubernetes docs, you may not change or remove an ephemeral container after adding it to a pod. If you make a mistake you must attach another one and live with the clutter until the pod is recreated. Ephemeral containers also cannot have ports, livenessProbe, readinessProbe, or resources, and are never automatically restarted.
Ephemeral containers attach to a running pod, so they work when the pod is up but misbehaving. When the pod is crash-looping there is no stable container to attach to, so use --copy-to to build a modified copy, with a new command, an added debug sidecar (--share-processes), or a fatter image (--set-image). The original crashing pod is left as-is for further inspection.
Photo by Bernd Dittrich on Unsplash
Key Takeaway
Distroless and CrashLoopBackOff pods block kubectl exec because they have no shell or are already dead. kubectl debug attaches an ephemeral container with its own toolbox image, and --target shares the app's process namespace. For crashing pods, --copy-to clones the pod with a new command or debug image, leaving the original untouched.
The first Kubernetes debugging lesson everyone learns is kubectl exec into a pod and poke around. The second lesson, learned painfully in production, is that the two pods you most need to inspect are exactly the two you cannot exec into: the one stuck in CrashLoopBackOff, and the hardened distroless one that ships with no shell at all.
I hit both on the same on-call shift. This is how kubectl debug and ephemeral containers get me a working shell next to a pod that refuses to cooperate, and the gotchas nobody mentions until you trip over them.
CrashLoopBackOff is not an error in itself. It is Kubernetes telling you a container keeps starting, exiting, and getting restarted, with an ever-growing back-off delay between attempts. The process you want to inspect only lives for a second or two before it dies, so by the time you type kubectl exec, there is no running container to attach to and the command fails.
Logs and kubectl describe pod are the right first stop for the exit reason. But when the crash depends on the live filesystem, the mounted config, or a DNS lookup that only fails inside the cluster, reading logs is not enough. You need to be inside a container that stays up long enough to run commands by hand, and that is precisely what a crash loop denies you.
Distroless images strip everything that is not your app: no shell, no package manager, no coreutils. That is a genuine security win, because it shrinks the attack surface and removes the very tools an attacker would reach for. The Kubernetes docs are blunt about the cost: since distroless images do not include a shell or any debugging utilities, it is difficult to troubleshoot them using kubectl exec alone. There is simply no sh for exec to launch.
# A distroless pod has no shell, so exec just fails:
$ kubectl exec -it api-7d9f -- sh
OCI runtime exec failed: exec failed: unable to start container process:
exec: "sh": executable file not found in $PATH: unknown
# Attach an ephemeral debug container instead — a full toolbox
# is added next to the running app, sharing its Pod:
$ kubectl debug -it api-7d9f --image=busybox:1.28 --target=api
# --target=api shares the target container's process namespace,
# so from busybox you can see the app's PIDs, /proc, and env.
# Swap busybox for netshoot when the problem is the network:
$ kubectl debug -it api-7d9f --image=nicolaka/netshoot --target=apiThe --target flag is the whole point. It shares the target container's process namespace with your debug container, so from busybox or netshoot you can see the app's PIDs, read its /proc, and inspect its environment even though the app image itself has no tools. Note that process namespace sharing must be supported by the container runtime, and not every platform allows it.
An ephemeral container is a temporary container that kubectl debug adds to a running pod through a special ephemeralcontainers subresource in the API, not by editing the pod spec. Because it rides inside the existing pod, it shares the pod network and, with --target, the target's process namespace. You bring your own image full of tools, so the app image stays minimal while you still get a full shell.
The official docs put it plainly: ephemeral containers are useful for interactive troubleshooting when kubectl exec is insufficient because a container has crashed or a container image does not include debugging utilities. That single sentence covers both of my on-call problems at once, which is why this is the first tool I reach for now.
Ephemeral containers are perfect for a pod that is running but misbehaving. They do not help when the pod is crash-looping, because there is no stable container to attach to. For that case kubectl debug can clone the pod with --copy-to: it builds a copy where you override the crashing container's command with a shell, so the copy boots and idles instead of dying. You can also add a debug sidecar with --share-processes, or swap a slim image for a fat one with --set-image. The original pod is left exactly as it was.
# 1) Copy the pod and drop a debug container next to the app,
# sharing the process namespace — app keeps its own command:
$ kubectl debug myapp -it --image=ubuntu --share-processes --copy-to=myapp-debug
# 2) The container crashes on boot, so nothing to exec into.
# Copy it but override the ENTRYPOINT with a shell so it stays up:
$ kubectl debug myapp -it --copy-to=myapp-debug --container=myapp -- sh
# 3) Swap a slim production image for a fat one with debug tools.
# '*' rewrites every container in the copy:
$ kubectl debug myapp --copy-to=myapp-debug --set-image=*=ubuntu
# The original crashing pod is untouched; you poke at the copy.Two ephemeral-container gotchas that bite in production: you may not change or remove an ephemeral container after adding it to a pod, so a typo means you attach another one and live with the clutter until the pod is recreated. And ephemeral containers may not have ports, so ports, livenessProbe, readinessProbe, and resources are all disallowed. They also lack resource guarantees and are never automatically restarted.
Both give you a command line inside the cluster, but they solve different problems. exec runs a program that already exists inside the running container; debug injects a brand-new container with the tools you need. Here is how I decide between them.
| Concern | kubectl exec | kubectl debug (ephemeral) |
|---|---|---|
| Needs a shell in the image | Yes — fails on distroless | No — you bring your own image |
| Works on a crash-looping pod | No — no live container to attach | Use --copy-to for a stable copy |
| Adds a new container | No — runs inside the existing one | Yes — a temporary debug container |
| Can be removed after use | Nothing to remove — just exit | No — stays until the pod is recreated |
My rule of thumb: reach for kubectl exec on a healthy pod that happens to ship a shell, jump to kubectl debug with an ephemeral container the moment the image is distroless or the tools are missing, and fall back to --copy-to when the pod cannot even stay alive. Keep a network image like nicolaka/netshoot handy, and you can chase connectivity problems without ever baking debug tools into production images.