Why only 5 of 62 invoices matched

The problem

An internal finance tool collects supplier invoices from email and documents, reads the key fields out of each PDF, and matches them against the payments in the business bank account, which come from the accounting system every night. The point is a simple answer to an awkward question: which payments have no invoice behind them?

In early July it had 62 invoices and had matched 5. The other 57 were stuck. Nothing had reported an error.

Asking the right question first

The first correction came before any of the bugs, and it was about direction. The dashboard counted unmatched invoices, which answers "which invoices have not been paid?". For keeping the books, the useful question runs the other way. A payment out of the bank is a known fact; what is uncertain is whether each one has a supplier invoice to support it. So the tool now starts from each payment and looks for its invoice, and the headline figure is payments with no invoice rather than invoices with no payment.

Why so little matched

Querying the data directly found the main cause quickly. 45 of the 57 stuck invoices had an invoice reference but no supplier name, and sampling them turned up three unrelated invoices, from different suppliers for different amounts on different dates, all with the same extracted reference: billing.

The pattern that pulled invoice numbers out of the PDF text looked for the word "invoice", a separator, and then a run of letters and numbers. It ran case-insensitively, so a heading like "Invoice - Billing Address" satisfied it perfectly and produced "Billing" as the reference. The reference signal alone is worth four-fifths of the score a match needs, and the word "billing" will never appear in a bank transaction. Every invoice it touched was blocked permanently, however good its other evidence.

The fix was deliberately small: look at every candidate the pattern finds, not just the first, and take the first that contains a digit. I considered a cleverer pattern and chose a separate filter, because a filter is easier to read and test than a regular expression that is trying to do two jobs.

That stopped new bad references, but did nothing for the invoices already stored with one, so a repair job re-read the affected PDFs. It only fills a field that is empty or holds a known-bad value, and never overwrites one that looks plausible. A second reading is not guaranteed to be better than the first, and corrupting a correct invoice to fix a broken one is worse than leaving both alone.

Then the repair ran, matching ran again, and produced no new matches at all. The repair had swapped "billing" for a different false reference: a fragment of a web address that happened to sit beside the word "invoice" on one supplier's template. It contained digits, so it passed. Every genuine reference in the data was capitals and numbers, so the filter now also rejects anything with a lower-case letter, rather than trying to list every shape a web address can take.

The same day turned up two more causes that had nothing to do with the first. An invoice whose date could not be read was skipped before scoring even began, with no log line, so it looked exactly like one that had not been processed yet. And receipts that the accounting system had already attached to a specific payment were thrown back into the general matcher to be guessed at, when the answer had been supplied.

The failures that looked like success

Matching was only the visible part. The same review found the rest of the tool reporting calm while it failed underneath.

• About 96% of all scheduled executions had been failing for a week after one deployment, from four independent causes: a library pinned to a version its neighbour could not load, an empty list arriving as null from upstream, an archive writer pointed at the wrong storage account, and a database that had never been given the schema change its code expected. Fixing one would have left three.
• The accounting-system client turned errors into empty lists. A rate limit or a server error during the nightly pull looked like a night with no new transactions, and the sync recorded success. List calls now throw. A single lookup that finds nothing still returns nothing, because a missing attachment is a real state.
• The archive's integrity check had never run. The database had columns for when each archived item was last verified, and the dashboard reported how many were waiting, but no code ever wrote to them, so the waiting count could only grow. A weekly job now re-reads archived items, oldest unchecked first, and checks each one against the hash recorded when it was archived.

What I would do differently

I would have sampled the stuck records before touching the matching rules. Five out of sixty-two reads like an algorithm that is too strict, and the natural instinct is to tune thresholds. The algorithm was mostly fine; it was being fed a word instead of a reference, and it sat among jobs that could fail while reporting success.

I would also treat "nothing to do" as a result that needs evidence. Every failure here presented as an absence: no reference worth matching, no date, no transactions, nothing waiting to verify. An empty result is the one a broken system produces most easily, so it is the one most worth checking.

Related

If this is your problem too

If a reconciliation or matching process quietly leaves most of its work undone, the diagnosis usually starts with twenty stuck records rather than the code. The system itself is described in the financial management application case study, and the integration behind its nightly sync in fifteen API routes that never existed.