Data engineering · Quality and governance
Knowing a number is wrong before anyone reads it
Checks on the data itself, lineage from source to report, named owners and a catalogue people actually use. Governance as a working practice for the people who maintain the data, rather than a framework written for an audit.
Where this starts
Everything can be running while the data is wrong
A report can load on time, from a platform that is fully available, and still show a figure that is wrong because a source sent half its rows or a code quietly changed meaning. Service monitoring cannot see this, because nothing has failed. The first detector is usually a senior reader who senses the number looks odd, by which time it has already been circulated.
What we usually find
- Errors in figures discovered by the people reading them
- No way to see which reports a broken source affects
- Datasets with no owner, so issues sit unresolved
- Definitions held in people’s heads rather than written down
- A catalogue built once and never opened again
- Governance meetings that produce minutes but not change
Our position
Governance that does not change what happens on an ordinary Tuesday afternoon is documentation. Ownership means a named person is told when their data breaks.
What the work covers
Checks, lineage and ownership
The regulatory control set belongs to governance and compliance under cybersecurity. This is the operational side: whether the data is right, where it came from, and who fixes it.
Quality checks
Freshness, volume, schema, uniqueness and validity checked on every load, placed at the points where a failure would otherwise travel furthest.
Anomaly detection
Unusual movements in key figures flagged automatically, catching the errors no rule anticipated, such as a region quietly dropping to zero.
Lineage
Every report traceable back to its sources and every source forward to what depends on it, so the impact of a problem or a change is known before it is felt.
Named ownership
An owner for each important dataset who is notified when it fails and holds the authority to fix it, rather than a role on a chart.
A catalogue people use
Datasets, definitions and owners searchable in one place, populated from the platform so it stays current without anyone maintaining it by hand.
Issue management
Data problems logged, prioritised and closed like any other defect, with the recurrence of each type tracked so root causes get fixed.
How quality is enforced
Bad data stopped at the gate
Checks that only report are ignored within a month. These are wired to act.
Circuit breakers
A failed critical check halts the downstream refresh, so reports keep showing yesterday’s correct figures rather than today’s wrong ones.
Routing
Failures sent to the dataset owner with the failing records attached, not to a shared channel nobody watches.
Status on reports
Each report showing the freshness and check status of its data, so a reader can see when a figure is provisional.
Scorecards
Quality measured over time by dataset and owner, so improvement is visible and persistent problems become obvious.
Catalogue sync
Metadata, lineage and ownership refreshed from the platform automatically, so the catalogue never describes an estate that no longer exists.
What you are left holding
- Checks on freshness, volume, schema and validity at every critical load
- Lineage from every source to every report
- A named, notified owner for each important dataset
- A catalogue populated from the platform itself
- Reports that show when their data is provisional
Tell us about the last wrong number that reached a meeting
Tracing how it got there usually shows exactly where the first check should sit.