The morning email that went quiet

The problem

Every weekday at 09:00 and 13:00 a scheduled function sends me one email: what is unread in the shared mailboxes, how the site did, what the ads cost and whether email authentication is healthy. What it is and how it is built is in the mailbox digest case study. This entry is about what happens when it does not arrive.

An error is easy to notice. Something throws, a log line appears, an alert fires. A scheduled email that never runs throws nothing. There is no failed request and no red line on a chart, only a morning without an email, and the only sensor is a person who was expecting one. Between June and October 2026 the digest failed that way, or came close, five times. Each had a different cause, and they all had the same shape: for a scheduled email, silence is the failure, and nothing alerts on silence.

Five quiet failures

The timer that stopped while the button worked. In June, as part of moving storage access off shared keys, key-based access was switched off on the digest's storage. The function host still used a key for its own bookkeeping, and the platform relies on that bookkeeping to coordinate scheduled triggers. When it failed, the schedule stopped firing while runs started by hand carried on working. The last scheduled digest went out on 19 June. A manual run on 24 June succeeded, which is exactly the evidence that makes a schedule look fine. Runtime telemetry had been switched off as well, so the errors that did exist were hidden. The fix, on 25 June, moved the host's own storage connection onto its managed identity.

The link that expired after a week. Each deployment told the platform to fetch the application package from a signed link, and the kind of link the pipeline could sign lasts at most seven days. Nothing broke on day seven. It broke at the next restart after day seven, when the platform tried to fetch the package again, failed, and never started the app. In July that meant five days with no digest at all, from 22 to 27 July. The fix landed and was reverted the same day, after it left production returning errors for ten minutes while it waited for a person to undo it. The work item was closed anyway, so the expiring link stayed, and production went dark four more times. Telemetry recorded zero failed requests across every one of those outages, because a host that never starts never sends any. The evidence was rows that were missing, not rows that were red. Every outage was found by someone noticing the emails had stopped.

The pipeline that would have sent everything twice. On 14 September the digest moved to a new hosting plan. The pipeline still pointed at the retired app, and the step it would run switched that app's timer back on. One approved run would have started a second host sending every digest alongside the first, and the approval prompt named only the build version, not where it was going. Nothing had failed, so nothing said so.

The section that disappeared. The email leaves out any section with no rows, which was meant to keep it short. A review of the database on 7 October showed what that cost: the Google Ads section had quietly vanished after 23 July. Whether that was a broken feed or paused campaigns, the email gave no way to tell, because it said nothing at all.

The check that told nobody. The same review found a second silence. An availability check had recorded “red” on every run since 22 September, 36 times in all, and not one of those results had been sent to anyone.

What I chose, and why

Each failure was the absence of something, and every piece of monitoring I had was built to notice the presence of something. So each fix turns an absence into a signal.

• Watch from outside. An availability test now polls the app from five locations and alerts when the host stops answering. A dead host cannot report its own death, and an outside probe does not care what time the schedule is due, so it cannot raise a false alarm outside the sending window.
• Remove the thing that expires. The package is no longer fetched from a signed link, so there is nothing left to run out. The fix that had been reverted in July went back in at the end of August, and the move to a new hosting plan in September retired the link altogether.
• Check the target before anyone is asked to approve. The pipeline confirms the destination is on the expected hosting plan and refuses the retired app by name, and the approval prompt now says where the release is going. After deploying, it reads the retired app's timer setting and fails if it has been switched back on, without ever writing to it. I tested the guard by reintroducing fifteen known mistakes one at a time. All fifteen were caught, and writing them exposed one check that was weaker than it looked.
• Print empty. The digest now opens with a data-health card. Each feed is judged against its own rhythm, from next-day feeds expected within 48 hours to weekly ones. It shows the availability status with when the current streak began and whether anyone was told, and it counts lost enquiries and testimonials nobody has moderated. When everything is current it collapses to one green line, so it costs nothing on a normal day. A feed whose query fails says “query failed” rather than taking the whole email down.

The release check on this same digest once failed in the opposite direction, reporting two good deployments as broken. That story, and why a passing health check proves less than it seems, is in a green health check is not a deployment.

What I would do differently

I would build the outside check on the first day. A twice-daily email is precisely the kind of system whose failure mode is silence, and a probe that expects to hear from it costs almost nothing beside five days without it.

I would also stop treating a merged fix as a finished one. The July fix was merged, reverted and closed, and with the work item closed, nothing was tracking the failure when it came back four times. A fix is finished when the outcome is observed, not when the change lands, and “it worked when I ran it by hand” is evidence about the code, not about the schedule.

Related

If this is your problem too

If you have a scheduled job that nobody would notice stopping, finding it is a short piece of work in a diagnosis. Another job that reported success while doing harm is the cleanup job that came for production, and the wider subject is production reliability.