Six weeks of ‘done’ that did nothing

The problem

This site keeps a small contact store for people who get in touch, with email addresses encrypted at rest. Once a day, a sync copies those contacts into an Outlook mailbox and reads back any contacts that were added there, so the two stay in step.

In early October a data protection impact assessment, reading the code rather than the logs, found that the sync sent the encrypted value to Outlook as the email address. Any contact it wrote would carry ciphertext where the address should be. Worse, the return trip would read those contacts back and treat the ciphertext as a new email address, minting identities for people who did not exist.

The first fix

The repair had three parts. Decrypt before sending, so Outlook receives the real address; an old unencrypted value passes through, and anything that is neither is left out rather than sent as junk. On the way back, recognise a value this system encrypted and use the real address, so a round trip creates nothing new. And re-send, once, any linked contact that Outlook holds with ciphertext in it, because a contact was only ever re-sent when it changed, so the scrambled ones would otherwise never be repaired.

One decision is worth showing. The question "is this value ours?" is answered by whether it decrypts with our key. An earlier draft took a shortcut: a value containing an @ must be a plain address. The ciphertext produced by the test setup contained an @, which was the end of that idea. The only reliable test for "this is our ciphertext" is the one that cannot be fooled by what the bytes happen to look like.

It shipped on 7 October. The repair would run at the next morning's sync.

Six weeks of “done”

It did not run. The work item said to close it only once a synced contact showed a readable address, so the next day I went looking for one, and then for the run that should have produced it.

Every daily run since the sync moved to a new backend on 25 August had done the same thing. It asked for the mailbox's contacts, was refused with a 403, logged a warning, and then logged that it was done, every one of its counts at zero. The application had permission to read and send mail, but not to read contacts. The service treated the refusal as an empty mailbox, and the timer that ran it swallowed exceptions, so even a louder failure would have been recorded as a success. For six weeks the job reported that it had finished, and it had done nothing at all.

What I chose, and why

A refused read now throws, and the timer logs the error and rethrows it, so the run is recorded as failed. A permission refusal is not an empty result; it is the absence of a result, and the code now says so.

Throwing, rather than quietly skipping the read, matters for the other half of the sync. Its second phase creates Outlook contacts for any identity not already linked to one. Run against an inbound read that failed, it would have no idea what Outlook already held, and would create duplicates of contacts that were already there. Stopping the whole run is the only safe answer when one half of it cannot see.

The missing permission was granted outside the code, and narrowly. The broad way would have been a tenant-wide grant letting the application read and change the contacts of every mailbox in the organisation. Instead it was granted through Exchange's role-based access for applications, scoped to the one mailbox the sync uses, and checked from both sides: in scope for that mailbox, out of scope for another.

Where it stands

The repair shipped on 7 October and the permission was granted on 8 October. The change that makes a refused read fail the run reached production a few hours before the next sync, which ran at 05:00 UTC on 9 October. It read the mailbox, linked one contact that already existed there, wrote three new contacts with readable addresses, and repaired nothing.

Nothing needed repairing. The missing permission that hid six weeks of doing nothing had also stopped the broken code from ever writing a contact, so no scrambled address reached the mailbox. The same gap hid one bug and contained the other. That was luck, not design, and it is why the definition of done asked for a real contact rather than a clean run.

What I would do differently

I would never let a scheduled job's success mean "it did not throw". A run that reads nothing, creates nothing and repairs nothing, every day for six weeks, is a signal in its own right, and a job like this should report zero work as something to look at rather than something to log.

The thing I would keep is the definition of done. The work item was not closed when the fix was released; it was to close when a real contact showed a real address. That is the only reason the six weeks came to light within a day of the release, rather than staying hidden behind a closed ticket.

Related

If this is your problem too

Scheduled jobs that report success are worth checking against what they actually changed, and that is a quick part of a diagnosis. Another 403 that turned out to be a permission verdict rather than a broken token is in fifteen API routes that never existed, and keeping permissions narrow is the subject of deleting the last connection string.