The cleanup job that came for production

The problem

A container registry fills up. Every build pushes an image, most images are never used again, and storage is billed by the gigabyte, so a scheduled purge that deletes old tags is ordinary housekeeping. I set one up and stopped thinking about it.

The purge deleted the image production was running on.

Why it happened

The purge selected by age. The running service selected by digest — pinned deliberately, so that a restart brings back the same code rather than whatever happens to be newest. Those two facts are individually sensible and jointly fatal: a digest that has been stable for weeks because it is the one in production looks exactly like an old tag nobody wants.

Nothing broke at the moment of deletion, which is the part worth sitting with. A running container does not re-read its image. The damage was invisible until the next restart, when there was nothing to pull.

The first fix, and why it was not enough

The obvious repair is to exclude the live digest from the purge, and that is what I did first: read the digest the service is actually running, and protect it.

That fix has a hole in it, and the hole is the interesting part. It assumes the check runs. If the query that discovers the live digest fails — a transient API error, a permission that lapsed, a renamed resource — the purge is left holding an empty list of things to protect. An empty exclusion list does not look like a failure. It looks like permission to delete everything.

What I chose, and why

The purge now fails closed. If it cannot establish what is running, it does not run at all, and it says so loudly rather than quietly doing nothing. A purge that skips a night costs a few pence in storage. A purge that runs on bad information costs a production image.

That asymmetry is the whole design. The two outcomes are not equally bad, so the code should not treat a missing answer and a negative answer as the same thing. Most housekeeping automation I have seen treats them identically, because the happy path is written first and the error path inherits whatever the happy path's variable was initialised to.

The general shape of this

This is not really a story about container registries. It is about a category of job that is easy to under-think: automation whose entire purpose is deletion, running unattended, on a schedule, against production. Backup pruning, log retention, stale account cleanup, orphaned resource sweeps. They all share the shape.

They are written when the system is small, when the blast radius is genuinely tiny, and are rarely revisited when it stops being small.
Their failure mode is silence. A deletion job that deletes too much produces no error. It produces a success.
They are trusted because they have never gone wrong, which is a statement about the past rather than about the design.

What I would do differently

I would have asked what happens when the safety check itself fails, at the point of writing the safety check. It took a second incident to ask it. The first fix was written in the frame of mind of "protect the live image", and the question that mattered was "what does this do when it does not know what the live image is".

I would also not have described the first fix as done. It was reported as resolved, and it was resolved for the case that had actually happened, which is a narrower claim than the one I made at the time.

Unattended jobs with delete permissions are worth an hour of anyone's attention during a diagnosis. The same pipeline's cost story is a pipeline that cost more asleep than awake, and the wider subject is production reliability.