Claude on Question 13, Sep 27

Why signals don't work

A zombie (state Z, shown as <defunct>) isn't a running process — it has already exited. All that remains is an entry in the kernel's process table holding its exit status. There's no code to execute, no memory mapped, nothing to interrupt. Signals, including SIGKILL, are simply ignored because there's nothing left to signal.

The zombie persists because its parent hasn't called wait()/waitpid() to collect the exit status. Fix the parent, and the zombie disappears.

Finding them

ps -eo pid,ppid,stat,comm | awk '$3 ~ /^Z/'

Or a quick count:

ps -eo stat | grep -c '^Z'

Getting the parent to reap

First, identify the parent:

ps -o ppid= -p <zombie_pid>

1. Nudge the parent with SIGCHLD:

kill -s SIGCHLD <parent_pid>

This works only if the parent has a functioning SIGCHLD handler that it somehow missed a delivery for. Usually it does nothing, but it's free and non-disruptive, so try it first.

2. Kill the parent.

kill <parent_pid>        # SIGTERM
kill -9 <parent_pid>     # if it ignores SIGTERM

When the parent dies, orphaned zombies are re-parented to PID 1 (systemd or init), which reaps them immediately. If the parent is a service, restart it properly instead:

systemctl restart <service>

When you can't kill the parent

If the parent is a critical process you can't restart right now, you're stuck — there is no way to force-reap a zombie from userspace. The kernel deliberately provides no such interface, since the exit status belongs to the parent.

The one workaround is attaching a debugger to call wait() on the parent's behalf:

gdb -p <parent_pid> -batch -ex 'call (int) waitpid(-1, 0, 1)' -ex detach

Treat this as a last resort: it runs code inside a live process and can corrupt its state or crash it, especially if the parent is multithreaded or in the middle of a syscall. Only reach for it on a process you're prepared to lose anyway.

Does it actually matter?

A handful of zombies is harmless — a few hundred bytes of kernel memory each. The real problem is PID exhaustion, since each zombie holds a PID slot:

cat /proc/sys/kernel/pid_max     # typically 32768, or ~4.2M on newer kernels

Steady growth in zombie count means the parent has a bug: it's forking without ever reaping. Common culprits are a missing waitpid() loop, or SIGCHLD set to a handler that doesn't drain all pending children. The correct pattern in the parent is:

while (waitpid(-1, NULL, WNOHANG) > 0)
    ;

Or set signal(SIGCHLD, SIG_IGN) to have the kernel auto-reap. Killing the parent is a mitigation; fixing or updating it is the actual repair.

Containers

Zombie accumulation in Docker/Kubernetes is nearly always a PID 1 problem. Your application typically runs as PID 1 but isn't written to reap orphans, so anything re-parented to it accumulates forever.

docker run --init ...    # Docker injects tini as PID 1

For Kubernetes, shareProcessNamespace: true gives the pod a reaping PID 1, or bake tini/dumb-init into the image:

ENTRYPOINT ["/usr/bin/tini", "--", "/app/start.sh"]

This is also worth checking if your entrypoint is a shell script — sh -c "..." as PID 1 won't reap either, and execing the real process doesn't help unless that process reaps.