A dashboard is only as honest as its labels

The problem

I built an internal dashboard so that I could stop opening five portals every morning. It reads the same database that this site's analytics, advertising and enquiry pipelines write to, and it shows the numbers I make decisions from.

Between early September and early October 2026 I found that several of those numbers were wrong in the most dangerous way there is. No query failed, but the label on the screen described something different from what had been counted. A chart that throws an error gets fixed. A chart that quietly answers a different question gets believed.

Five ways my own dashboard lied to me

• Paid traffic counted as organic. The database had been recording the identifier Microsoft attaches to an ad click for weeks, but the application's description of that table had no field for it, so Microsoft paid visits were filed as organic. Over 120 days, counting them properly moved paid traffic from 33% to 42% of attributed sessions, more than doubled paid booking intents from 23 to 52, and cut the cost per paid booking intent by more than half.
• A stopped feed shown as a quiet month. The Google Ads data had no rows since July and Search Console none since June, and both pages said there was no data for the period. That reads as “you had no activity” rather than “this feed stopped weeks ago”, and the two call for opposite next steps. The home page's “Data ingestion: 100%” card looked like the dashboard's own health light, but it measured one pipeline, the email-authentication reports.
• Percentages on a base of one. Contact submissions going from 1 to 3 read “+200%” with the same weight as a move across thousands of sessions. The same component printed a fall without its minus sign, so anyone reading the number rather than the colour and the arrow read a drop as a rise.
• One word, two meanings. Until 22 September 2026 a “conversion” was a click on a call to action; from that date it is a completed contact form. The dashboard added the two together, so any date range spanning the change reported a collapse in conversions that never happened.
• A sample presented as a total. The session and visitor tables hold only visitors who agreed to marketing consent: about 6.5% of sessions, 86 of 1,328 between 8 September and 7 October 2026. The dashboard presented them as site totals, and the conversion rate divided by them, so it read roughly fifteen times too high.

146 enquiries, 3 accepted

Thirty-two of the 67 tables in the analytics database were not wired into the application at all, and one was the only record of contact-form submissions that never became an enquiry. Once it was on screen, the leads page showed that in the 30 days to 7 October 2026 the contact form received 146 submissions and accepted 3, about 2%. Read as a funnel, either the form is broken or 143 people were turned away. It was neither, but the rejection labels were too coarse to show it.

• 65 were rejected as requests the server could not read. That looks like a defect in the submit path, but over the same 30 days real browsers recorded three attempts to send the form and not one validation failure. These requests were scripted, sent straight at the endpoint without the page, and this was the one rejection path logged without the abuse signals every other path records. It now records them too.
• Most of the rest failed the check that a submission came from this site, which as one number reads like a broken allow-list. The application logs showed 95 rejections at that check, 91 of them from regions the site does not serve. Only 3 were genuine mismatches, and the address that failed had never been logged, so they could not be judged either.

Blocked regions now have a reason of their own, and a genuine mismatch names the host that failed, never the full address. A test comparing the browser's list of reasons with the server's found the two had already drifted: one reason was filed under a different stage in each, and the server's version is the one the daily counts and the morning digest read. The leads panel shows the two checks side by side rather than as a funnel, because they are independent and the data does not record which ran first. Drawing them as a sequence would claim an order nobody measured.

What I chose, and why

One rule: every figure says what it counts, and when no label can make a figure honest, the figure is withheld rather than shown with a caveat.

• Label at the point of display where the number is right but partial. Everything drawn from the consented sample now says “consented only”, and the conversion rate reads “per consented session”.
• Cut rather than label where the meaning changed. I considered marking the older conversion figures and rejected it: a count of clicks beside a count of forms is not a comparison any label can fix. Conversions are now counted from 22 September 2026 only.
• Keep the movement, drop the ratio where the base is small. Below 20 in the previous period, the absolute change still shows, without a percentage.
• Break silence only where silence misleads. Each source shows when its data last arrived and how far past its normal rhythm that is, and nothing at all when it is healthy.

That last fix needed fixing within a day. The Google Ads feed I had just flagged as stale was fine: all three campaigns were paused, and an ad platform produces no rows whenever spend stops. So the health page now knows three kinds of source. A scheduled feed going quiet is a fault, an activity table going quiet may be a slow week, and a campaign going quiet may be a business decision. Only the first raises an alarm.

What I would do differently

I would compare the application's description of the database with the database itself before trusting either. The costliest error came from exactly that drift, and nothing failed when it happened.

I would also write the label before the query. Deciding what a figure is called forces a decision about what it counts, and most of these five would not have survived someone asking “of what?”

Related

If this is your problem too

If the numbers your business runs on have never been checked against what they count, that is a short piece of work in a diagnosis. The pipeline these figures come from is described in the first-party analytics platform, and what happens when you press send on the contact form is in how this site actually works.