Data freshness and history

A figure that looks wrong is usually one of four things, and they are distinguishable. This page is how to tell them apart before concluding the data is broken.

The four causes, in the order worth checking

1. The screen filtered it out. The filtered transaction screens default to roughly the last three months and a row limit. Data outside that window is present and simply not shown. This is by far the most common cause and costs nothing to rule out: widen the window.

2. A refresh has not run yet. Each step of the pipeline runs on its own schedule. A transaction can be imported but not classified, or classified but not calculated. The record exists; its margin does not yet.

3. A step is switched off. Whether a step runs is a database switch. A step that is off leaves its data getting older with no error anywhere, which is exactly what the healthchecks exist to surface.

4. The history was never there. Some periods have no data from any source, and some have data from a different source than the period beside it. That is a fact about provenance, not a gap to be repaired. Every object document carries a provenance table saying which period came from where.

The banners

Five counters appear above every screen. They show to everyone regardless of permission, so a link in one may refuse you; the count is still the point.

A backlog banner is not an error. It means work is in progress, and the figure you are looking at may move.

Slow pages

Banner queries run before the page content, including on documentation pages. The healthcheck summary counts both realised and expected impact queues; the separate sales-order banner counts the realised queue again. On a large backlog these counts can dominate page time. A slow page therefore needs timing of the shared queries as well as its own controller, not an assumption that its content is slow.

The healthchecks

Each monitored step has a check that ages its last successful run against an allowance. A daily step is allowed 25 hours, one hour of slack on a 24 hour cycle. The realised and expected sales-order queue checks count unprocessed rows against a threshold rather than ageing anything, because those queues have no last-finished moment.

A paused step still ages. Its check goes red once it passes its allowance, which is correct: the data really is getting stale, whatever the reason.

The case that hides

A stale customer dimension leaves margin correct. The customer reconcile is not an input to margin, so CM1 and CM2 are produced as usual while customer grouping and order sequence quietly fall behind. Nothing on a margin screen indicates it. The only signal is that check going red.

What a green check does not prove

A finished batch means the job completed, not that every row was written. The database is strict about over-long values and invalid dates, and the code does not check for errors on every write, so a row can be silently skipped inside a run that reports success.

Reading a real row back is the only check that catches this. A green batch is not evidence.

Related

• The pipeline
• Missing or stale data
• Queues and crons
• Checks themselves: /system/healthchecks and /system/crons

Evidence

Banner set and their sources from templates/index.tpl and public/index.php. Check kinds, the 25 hour allowance and the queue check from the healthcheck blocks in config/crons.json and app/models/healthchecks.php. Pause semantics and the "paused still ages" behaviour from app/models/batches.php and AGENTS.md. The silent-write failure mode from AGENTS.md, What fails silently. Basis: code-checked against the current implementation.