While a Pod That Returns Healthy Responses Kills the Traffic

A pod whose readiness probe returns 200 but then floods 500s once it starts receiving traffic is more common than you'd think. The probe passed, the pod is Running, and the replica count is normal. Nothing looks wrong on the dashboard, yet only users see errors. This gap opens up when you define health checks at the level of "is the process alive?"

Kubernetes readiness and liveness sound similar but do completely different things. Readiness decides whether to put this pod into the service endpoints, and liveness decides whether to kill the container and restart it. The first trap emerges here. If readiness fails, the pod is only removed from traffic; no restart happens. Conversely, if liveness fails, a restart happens, but being removed from traffic isn't guaranteed. The moment you wire both probes to the same handler, this distinction disappears.

Alive and ready are different questions

The most common anti-pattern is creating a single /health and attaching it to both liveness and readiness. This handler usually only checks that "the HTTP server responds." If the process can accept requests, it returns 200. Even if every dependency is dead, it's 200. The pod is treated as ready, and the service pushes traffic toward dead dependencies.

This leads to the reflexive conclusion: "Then I'll just put a DB ping in readiness." That's only half right. Adding a dependency check to readiness does remove an unready pod from traffic. The problem is what comes next. If the DB slows down briefly, every pod's readiness fails at the same time, all endpoints go empty, and the entire service goes down. That's a worse outcome than a restart, because it isn't one pod that's broken—it's all of them dropping out at once.

If you're going to include dependency checks, you have to design two things together. Put a short timeout on the check (1–2 seconds), and don't reflect failures immediately—require a number of consecutive failures. Setting failureThreshold to 3 absorbs transient latency. And even if readiness fails, liveness must never look at the same condition. It's safer to leave liveness as a way to check whether the process itself is deadlocked.

Liveness doesn't fix the problem

A liveness failure triggers a restart. A restart only resets state; it doesn't remove the cause. If you tie liveness to a health check that fails because an external API is down, the pod will repeatedly die and come back. Backoff widens the restart interval, it moves into CrashLoopBackOff, and the deployment never finishes. In a rolling update, if a new pod can't become Ready, you expect the old pods to remain according to maxUnavailable, but with a bad configuration there are moments when available capacity hits zero.

What's worth putting in liveness is things like a deadlock inside the process, a stalled event loop, or an unexpected runtime state. Limit it to signals you can judge from your own process alone. DB, cache, and external APIs should be excluded here. If you're unsure, use this criterion: "If this condition fails, does a restart actually solve the problem?" If it doesn't, it's not liveness.

What to use as a deployment gate

Whether rolling or blue-green, the key is what you check before sending traffic to the new version. Passing readiness is only a minimum condition, not a gate. Keep the following as candidates and choose according to your team's situation.

• Error rate after the new pods have handled real requests (5xx ratio, 5-minute window)
• p95, p99 of the latency distribution (the average hides things)
• Connection success rate to critical dependencies
• Smoke test results for features that exist only in the new version

Of these, I prefer error rate and p99 latency as the default gates. The reason is simple: both tie directly to user experience and are easy to judge automatically. Smoke tests cover too little, so passing them isn't reassuring.

Blue-green is structurally advantageous for applying this gate. The old version environment is still alive, so if the gate fails, you just send traffic back. In exchange, you have to weigh in advance the cost of keeping two environments running at once and the reversibility when you've changed a shared resource like the DB schema. If you only extended the schema, rollback is easy, but if you dropped or renamed a column, the old version breaks. In that case, rollback is effectively impossible and only forward fixes remain.

Rolling deployment, by contrast, is slow to reverse. New pods have already received some traffic, and even if you roll back to the previous ReplicaSet, the data or side effects accumulated in the meantime remain. So in rolling deployments you set the gate more conservatively. Start with a low ratio of new pods and take a long observation window.

When to declare a rollback

The hardest decision is "when to roll back." A common mistake here is holding out to roll back only after understanding the cause. You can do root-cause analysis after the rollback. Set the criteria as numbers before deployment, and once that line is crossed, roll back without discussion. For example, codify a condition like "roll back if 5xx exceeds 1% and lasts more than 2 minutes."

If you deployed an irreversible change alongside, the situation changes. In that case, set the criterion as a forward fix rather than a rollback, and spend your resources on narrowing the blast radius. Either way, it's too late to create the criteria during the deployment. Only the numbers decided before deployment can be trusted.

Verification is also possible outside the deployment pipeline. It's a method where you stand up a single new-version pod and route a portion of real traffic to it, measuring actual response quality rather than readiness. This approach is useful in that you can observe how readiness reacts when dependencies are slow and how that affects overall availability. When changing probe settings, it's safer to verify under real traffic conditions first and then apply them.

Ultimately, what you decide first in health check design isn't the probe's path but who owns the failure. A readiness failure should end with removal from traffic, and a liveness failure should only be wired to problems that a restart actually solves. Starting with your next deployment, begin by separating the two probes from the same handler and writing your rollback conditions down as numbers before you deploy.

Comments

Loading comments.