200 OK doesn't protect your zero-downtime deployment

It's common to see 5xx errors spike only about twenty minutes after a deployment finishes. The deployment dashboard is all green. Every pod shows Ready, and health checks passed. There's no red text in the logs. Yet the cause of the outage is clearly this deployment. They all share one thing: the health check returned a perfunctory 200.

The key to zero-downtime deployments isn't "the server never goes down" but "an unprepared server never receives traffic." Readiness is what decides whether a server is prepared. When that check becomes a formality, rolling and blue-green both end up the same way. The moment you trust the green signal and route traffic in, the outage is already booked.

These failures usually look alike: a health endpoint that returns hardcoded JSON from a single controller, one that returns {"status":"ok"} as long as the process is up. It has nothing to do with what the service actually depends on. Before traffic is routed in, there's no way to know whether that 200 is real, and the problem is created by the side that trusts it.

"Alive" and "ready to receive traffic" are different checks

There are two kinds of health checks. Liveness checks whether the process is alive and restarts the pod if it's judged dead. Readiness checks whether the pod can receive traffic and removes it from the service rotation on failure. Restart versus removal—completely different purposes. Yet surprisingly many teams use the same endpoint for both checks. As long as the process is running, the pod stays in rotation, whether the DB connection pool never filled up or the cache is completely empty.

The moment a framework binds the HTTP server is not the same as the moment a service can fully digest a request. Most default health endpoints only check the former. A check for the latter has to be built by hand.

What to put in readiness, what to leave out

What readiness should test is "can this pod handle a single request from start to finish?" Whether DB, Redis, and message queues should be included is a judgment call for each team. One useful criterion: "if that dependency died, could this pod still handle requests normally?" For a service where every request touches the database, a readiness check that returns 200 while the DB is down is worthless. On the other hand, if a read-only cache dying takes the pod out of rotation, traffic piles onto the remaining pods and that load pressures the cache to rebuild again. Cascade failures start here.

The implementation isn't hard. Put one dependency check with a timeout in the /ready endpoint. Rather than a perfunctory query like SELECT 1, it's better to pick the minimal path a service actually goes through, and return 503 if there's no response within two seconds. Kubernetes treats an httpGet probe response of 400 or above as a readiness failure. That single line acts as the gate for traffic switching.

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 3

When the check interval creates an incident

Detection latency is the product of the check interval and the failure threshold. With a periodSeconds of 10 and a failureThreshold of 3, it takes up to 30 seconds to pull traffic off a pod even after it goes bad. During those 30 seconds, the router keeps sending calls to that pod. Shortening the interval to 2 seconds creates the opposite problem. With 50 pods, the probes alone put dozens of requests per second of load on the dependency. It's manageable when things are healthy, but the moment a dependency wobbles, every pod's probe fails at once and makes things worse.

So the interval is a tradeoff between detection latency and probe load. Readiness at 5 seconds, liveness at 10 seconds, and a threshold of 3 is a solid starting point. Services that take a long time to start should get a separate startupProbe. If you put a liveness check on an application whose JVM takes 40 seconds to come up, the pod falls into a restart loop during warm-up. The standard practice is to keep liveness and readiness dormant until the startupProbe succeeds.

Why it blows up after the deployment is done

Why does the outage appear long after the deployment instead of right after? Because during deployment the pods have barely received any real traffic. A health check tests "is it okay to receive traffic right now," not "does it process traffic correctly once it's received." Situations like a column the new version needs not yet being migrated, or a newly connected external API slowing down, don't show up in process-level checks. Only after traffic is routed in does the connection pool fill up, cache misses accumulate, and timeouts stack on each other.

This is where deployment strategies diverge. Rolling deployment routes traffic to batches as each one passes readiness. If that readiness is a formality, broken code rides along from the very first batch and the rest follow. No matter how tightly you tune maxSurge and maxUnavailable, those numbers are meaningless if the check is a formality. Blue-green is fast because the switch itself is instantaneous, but the green environment sits idle until the switch. The verdict is reached without a single request of real traffic. It's safer to flow synthetic traffic into the green environment before switching, or to insert a canary that burns a small amount of real traffic first.

Don't decide on a rollback during an incident

In incident response, the most expensive phase isn't detection—it's judgment. Detection signals are already pouring out of the service. The problem is that "how bad is bad enough to revert" gets decided in the middle of the incident. At that moment the decision gets delayed, and while you delay, the new version grows the damage. Just write the numbers down before deploying.

• Roll back if the 5xx rate exceeds 3x the baseline for 3 consecutive minutes
• Roll back if p99 latency stays above 2x the baseline for 5 minutes
• Roll back if error logs that only appear after the new version's deployment spike

These numbers only mean anything if you know the normal margin of error. If a service already produces 0.3% 5xx errors normally, 0.5% isn't a warning sign. Before setting thresholds, pull the average and variance of a month of metrics first.

A rollback isn't always the right answer. If a data migration has already been applied, reverting to the previous version breaks things even harder because the schema no longer matches—the old version can't read data written by the new one. In that case, a forward fix is the right move, and separating migrations from the deployment unit lets you avoid that judgment call entirely. Rollback is a button you press after confirming "can we get away with reverting just the code?"

Starting with the next deployment, put a dependency check with a 2-second timeout into that single /ready line, and write the rollback trigger numbers into the deployment runbook. And go as far as injecting dependency failures in the test environment to verify those numbers actually fire. A perfunctory 200 doesn't protect your zero-downtime deployment. It just delivers the incident right on schedule.

Comments

Loading comments.