A pipeline that cost more asleep than awake

The problem

A compliance pipeline collects the daily email authentication reports that mail providers send (DMARC reports on mail sent in a domain's name, and TLS-RPT reports on encrypted delivery that failed), archives them, analyses them and raises alerts. The work is bursty: a lot of activity for a short while when reports arrive, then nothing for hours. It was running on a container host that stayed awake regardless, so most of what was being paid for was the host waiting.

Why it mattered

Not because the absolute figure was large. It mattered because the shape was wrong, and a cost with the wrong shape grows with the wrong thing. Paying for idle time means the bill tracks how long the host is switched on rather than how much work arrives, so it would have carried on rising with time rather than with volume, and no amount of tuning the processing would have touched it.

What I looked at

Azure Cost Management actuals for the resource group, at monthly granularity. The resource group is dedicated to this pipeline, so nothing had to be apportioned between workloads and the figure is the figure.

The trend mattered more than any single month: £12.31 in May, £18.68 in June, £23.16 in July, £26.83 in August. That is a cost climbing steadily while the amount of actual work stayed roughly flat, which is the signature of paying for availability rather than for processing.

The options

Leave it and accept the cost. Defensible at this size, and the honest baseline against which anything else has to justify itself. Rejected because the cost tracked time rather than volume, so it would keep climbing.
Shrink the always-on host. Cheaper, simpler, no processing changes. Rejected because it treats the symptom: a smaller host that is awake all the time is still paying for the wrong thing, and it reduces headroom for the bursts, which is the one moment the pipeline actually needs capacity.
Move to consumption compute with no always-ready instances. Pays for work rather than for waiting. The catch is that it changes the correctness problem, not just the billing one.

What I chose, and why

Consumption compute, scaled to zero. But the interesting part is the order, and the order was forced by a constraint rather than chosen for elegance.

Scale-to-zero and at-least-once delivery together will cheerfully process the same report twice. A host that comes and goes gets interrupted mid-run, and a queue that guarantees delivery at least once will hand the message to the next instance that wakes up. So the processing had to become genuinely idempotent first: each message claimed atomically, any interrupted run recovered at startup before new work is taken, and deployed images pinned by digest so a restart brings back the same code rather than whatever is newest.

Only then was it safe to move the host. The cost reduction was the last step, not the first, and describing it the other way round would make it sound like a billing change when it was a correctness change that happened to save money.

Where it got to

Running cost went from £26.83 in the last full month on the old host to a rate under £3 a month on the new one, doing the same work.

Those are Azure Cost Management actuals for a dedicated resource group. The post-migration rate covers nine days rather than a settled month, and I would rather say that than round it into something tidier. It is a measured rate, not Azure's own forecast.

The delivered system is written up in more detail as the Security Compliance Monitor.

What I would do differently

I would have looked at the cost shape earlier. The pipeline ran for months before anyone asked why an idle service was getting more expensive, and the answer was visible in the first cost query I eventually ran. Nothing about the fix required the delay.

I would also be more careful about how this gets retold. "Cut running costs by about 90%" is true and is nearly useless, because it invites the reader to expect the same from a workload with a different shape. The saving came from a specific mismatch between bursty work and always-on billing. Where that mismatch does not exist, neither does the saving.

Cost that tracks time rather than volume usually shows up during a diagnosis rather than being the thing anyone called about. Related reading: production reliability.