A green health check is not a deployment

The problem

Three times in five weeks, a deployment pipeline told me something about production that was not true. Twice it said a release had worked when it had not, and once it said two releases had failed when both were fine. No pipeline was broken in the usual sense: every step did what it was written to do, and every check passed or failed honestly against the thing it measured.

The trouble was what each check measured. A pipeline can make three quite different claims about a release, and I had been treating them as one.

• Deployment success. The platform accepted the package.
• Service health. Something at the address answers.
• Build identity. The thing answering is the build you just made.

Three incidents, one confusion

The release that went somewhere else. Between 30 August and 2 September, the pipeline for an email-report processing service reported one successful production deployment after another. It was deploying to a Container App that had been scaled to zero replicas after a migration to a different hosting plan, while the service that actually ran was still on a build deployed by hand at the cutover. Every signal passed: the status said Running, hourly summaries were being written, the scheduler answered. The new code was never live, but a database migration was, so the database was enforcing a rule the running code did not know how to handle.

The releases that failed for no reason. Later that month, two deployments of a mailbox digest both installed correctly and both failed their health check. I put the first down to a slow cold start and raised the waiting time from four minutes to fifteen. That diagnosis was wrong. The second failed the same way, and its improved error message printed the address it had been polling: https:///api/health. There is no host in it. A command-line tool update had changed where it put the app's hostname, the lookup came back empty, and the gate spent fifteen minutes proving that a blank address does not respond. The app had been healthy within seconds both times. Worse, the failure message advised rolling back, which would have replaced good code with older code because a gate was wrong.

The endpoint that stopped answering. At the end of September, an unsubscribe service stopped answering on its main route after a release, because a new component created a loop in how the application assembled its parts, and building that part of the application hung. The health endpoint never touched that part, so it stayed green. Two pipeline runs passed because the previous instance was still answering; their own health check reported the old commit, and nothing compared it. One run had failed on the hung route and gone unnoticed. The feature was not yet in use, so nobody was turned away, but the pipeline was no help in finding out.

What I chose, and why

Each service now reports what it is. A small build stamp written into the package at build time carries the commit and build number, and the service returns it on request. The package's own hash cannot live inside the package it is a hash of, so the deployment step supplies that as a setting and the service reports it back. The pipeline then fails if the reported commit or package hash does not match what it meant to deploy.

The detail that matters most is how absence is reported. A field the service cannot resolve is the literal word unknown, never empty, because a check that compares an empty expected value with an empty reported value passes on a service that knows nothing about itself. A service that cannot say what it is fails the check. Reading the stamp never throws either: a diagnostic that can take the service down is a bad trade.

Health checks stayed, labelled as what they are. They now print the address they are about to check, stop at once if the address is empty, wait against a deadline rather than a count of attempts, and say plainly that the package did deploy before suggesting anything as drastic as a rollback. The unsubscribe pipeline also waits until the health endpoint reports this build's commit, and a new test builds the application's real set of parts so the next loop fails in testing instead of production.

The same three questions apply to this site's own API deploys: is the host running, did the latest deployment finish after this one started, and does the build stamp name this run's commit? Asking only the first is what let every one of these incidents through.

What I would do differently

I would build the identity check first, not after an incident. A health endpoint costs almost nothing to extend with the commit it was built from, and without that, every green light in a pipeline is answering a smaller question than the one being asked.

I would also read the evidence before the explanation. A slow cold start was plausible for that mailbox digest, and I acted on it before looking at the address the gate was polling. The fifteen-minute window survived the correction, because a real cold start can still be slow, but it was not the fix.

Related

If this is your problem too

If you are not sure your deployments run what they say they run, that is a short piece of work in a diagnosis. Another job that reported success while doing harm is the cleanup job that came for production, and the wider subject is production reliability.