Turning Complex Problems Into Confident Technology

Deleting the last connection string

Security and secrets · Production · Inside the Workshop

The problem

Every service in the estate started life with a connection string or an account key in its configuration. That is how the tutorials are written, it works immediately, and it is the fastest route from nothing to a running system.

It is also a long-lived credential sitting in configuration, valid from anywhere, that nobody will rotate and no audit will notice until it is copied into somewhere it should not be. Over four services and about nine months, they have been removed one at a time in favour of managed identity.

Why one at a time, and not one project

Because it is not a refactor, it is a series of unrelated small negotiations with whatever each service happens to talk to. Storage, tables, queues, SQL and a mail API each have their own way of accepting an identity, their own failure mode when it is wrong, and their own set of role assignments that must exist before the code will start.

A big-bang migration would have coupled five independent risks into one deployment, for no benefit — nothing about doing them together makes any individual one safer.

What it actually costs

The code change is genuinely trivial: construct the client with a credential object instead of a string. Everything else is where the time goes.

  • The failure mode moves from startup to first use, and gets vaguer. A wrong connection string fails immediately and says so. A missing role assignment fails when the first call is made, with an authorisation error that does not name the role it wanted.
  • Some clients need the service URI instead of a connection string, so configuration that used to be one value becomes several, and the code has to choose a construction path based on which is present. That branch is easy to get subtly wrong and hard to notice, because the connection-string path keeps working.
  • SQL is its own exercise. Acquiring a token explicitly turned out to be more predictable than relying on the driver's own authentication mode, and the deployment identity needs database-level rights that the runtime identity must not have.
  • One service could not use a globalisation-trimmed runtime at all, because the SQL client cannot open connections under it. A setting adopted for image size quietly made identity-based database access impossible.

What I chose, and why

Managed identity everywhere it is supported, with role assignments provisioned as infrastructure rather than clicked, and the deployment identity's rights kept separate from the runtime identity's. Where a platform still requires a secret, it lives in a key vault and the application holds a reference rather than a value.

One warning learned the hard way and worth repeating: rotating a secret behind a key vault reference is not picked up by restarting the application. It needs a change to the setting itself. A restart looks like the obvious remedy and leaves the old value in place, which is a genuinely nasty way to spend an afternoon.

What I would do differently

I would start services on managed identity rather than migrating them to it. The work is roughly the same either way, and doing it first means the role assignments are part of the initial infrastructure rather than something retrofitted around a running system.

I would also stop treating "no secrets in configuration" as the finish line. The credential is gone, but the permission it represented still exists, attached to an identity — and an over-broad role assignment is the same risk wearing better clothes. The useful question is not "where are the secrets" but "what can this thing do, and why does it need to".

Related

If this is your problem too

Long-lived keys in configuration are one of the most common findings in a diagnosis, and usually the cheapest to fix. The least-privilege side of the same argument is in proving you honoured an unsubscribe. Related reading: production reliability.