The build agent I killed, then rebuilt

The problem

Azure DevOps gives a free tier of 1,800 hosted build minutes a month. In August 2026 this project used 1,807 across 151 runs. Going over is not catastrophic — it costs about $40 a month for another parallel job — but it is the kind of bill that arrives because nobody looked, and the free self-hosted option offers unlimited minutes in exchange for running the machine yourself.

The first attempt, which failed

In May 2026 I stood up an Ubuntu VM, installed the agent, and pointed the pipeline at it. Over about 32 hours it ran 26 jobs and 8 of them failed. Total Azure cost for the experiment: £1.64. I switched the pool back to ubuntu-latest and deleted the VM.

The failures were not mysterious, and none of them were about capacity:

The toolchain was incomplete. The agent had git, Node, Docker, Python and the Azure CLI. The pipeline needs the .NET SDK, PowerShell and a browser for the end-to-end suite. A hosted image comes with all of that pre-installed, and that is most of what you are paying for.
Docker left root-owned files in the workspace, which broke the next run's checkout. On a hosted agent this is invisible because the machine is destroyed after every job. On a persistent agent, yesterday's job is today's environment.

That second point is the real lesson and it is not about tooling. A hosted agent is stateless by construction. Moving to a persistent one silently converts every job into something that must clean up after itself, and nothing in the pipeline had ever been asked to.

What changed the second time

In September 2026 I rebuilt it, with the failures above as the specification rather than as surprises. The machine is provisioned from infrastructure-as-code with the full toolchain, and — the part that makes it worth having — it starts on demand and deallocates when idle, so it costs nothing while nothing is building.

It is running now, and at the moment of writing it is deallocated, which is the intended state rather than a fault.

What it actually cost

The honest accounting is not flattering to the idea. The first attempt cost £1.64 in compute and considerably more in attention. The rebuild cost a day of work plus three small fixes that only surfaced on the machine itself — a package that no longer exists in the base image, an install script that assumed the wrong shell, and a smoke test calling unzip --version, which unzip does not support.

Against roughly $40 a month for a hosted parallel job, this pays back slowly. It was worth doing because the build is also the place I test the things I advise other people about, and an agent I control is a better place to be wrong. It would be a poor trade purely on price.

What I would do differently

I would have written down the exact toolchain the pipeline requires before provisioning anything, rather than discovering it one failed job at a time. The list took twenty minutes to produce afterwards.

I would also not have deleted the VM. Keeping it deallocated costs almost nothing and preserves the evidence; by the time I came back to the question, the build logs from the failed trial had aged out of retention and I was reconstructing what happened from commit messages. My own notes recorded the machine as permanently gone, which was true when written and quietly false four months later.

"Should we self-host the build agents" is a question with a real answer that depends on your pipeline rather than on the pricing page. It usually comes up during a diagnosis. Related reading: production reliability and a pipeline that cost more asleep than awake.