Concept · No. 037 · November 22, 2026
A liveness probe that checks dependencies restarts healthy pods
A health check that pings the database looks responsible until the database slows down and the orchestrator restarts every replica at once. Liveness and readiness answer different questions, and mixing them turns a partial outage into a total one.
The idea
An orchestrator asks a running instance two separate questions. Liveness asks whether the process is broken in a way that only a restart will fix. Readiness asks whether it should receive traffic right now. Kubernetes makes the split explicit with livenessProbe and readinessProbe, and most load balancers and supervisors have some equivalent.
The problem it solves is real: processes deadlock, leak, or wedge, and something outside them has to notice and act. The trap is that a restart is a heavy action with a narrow purpose. It fixes local state. It does nothing for a slow database, a throttled API, or a partitioned network. A probe that fails for those reasons triggers a remedy that cannot work.
An example
A service exposes /healthz, which runs SELECT 1 against the shared Postgres primary, and the same endpoint backs the liveness probe.
- The primary fails over, and connections hang for longer than the probe timeout multiplied by the failure threshold.
- Every replica fails liveness at roughly the same moment, because they all share the same dependency.
- The kubelet kills all of them. In-flight requests are dropped.
- New pods start cold, with empty caches and empty connection pools.
- The new primary comes up and is hit by every replica opening its full pool at once, plus a backlog of client retries.
- The database slows under that load, probes fail again, and the cycle repeats.
Without the probe, the service would have returned errors for the length of the failover and recovered on its own. With it, the service has no capacity at all and is actively hurting the dependency it is waiting on.
Where it breaks
Readiness has the same failure in a milder form. If readiness depends on a shared dependency, every instance goes unready together, the load balancer has no endpoints, and clients see connection failures instead of a fast 503 or a cached response you could have served.
The opposite mistake is a liveness check so shallow it proves nothing. A handler on a separate thread that always returns 200 will keep a process alive while its worker pool is deadlocked.
Timeouts are a quieter hazard. Under heavy load, GC pauses or CPU throttling can delay a probe response, so liveness starts killing instances exactly when you need the capacity most.
In practice
I would start by writing down, for each probe, what action it triggers and whether that action can fix the condition it detects.
- Liveness checks process-local progress only: a worker loop's last-tick timestamp, or an event loop heartbeat. No network calls.
- Run the liveness handler on the same loop or pool that does real work, so a blocked loop actually fails it.
- Readiness reflects local state: config loaded, caches warmed, not draining. Handle shared dependency failures in the request path with timeouts and a circuit breaker that fails fast.
- Use a startup probe for slow boots so liveness never kills an instance that is still warming.
- Set liveness thresholds well above your worst observed pause. It should fire rarely, and only for a genuine wedge.
- On
SIGTERM, fail readiness first and keep serving in-flight requests until they finish. - Alert on restarts that happen across many replicas at once. Correlated restarts usually mean a probe is coupled to something shared.
- In staging, make the database unreachable and watch. Pods should stay up and return fast errors, not restart.
A liveness probe should only fail for conditions a restart can fix, so it must never depend on anything outside the process.