Data engineering · Master data and entity resolution
One record for each customer, however many systems they appear in
We match and remove duplicate customers, products and suppliers across your systems, and create one trusted record for each. We also put a process in place to stop the duplicates coming back next quarter.
Where this starts
Removing duplicates, again and again
Duplicate records are rarely a one-off problem. They are the steady result of a process. A customer is created again because a search did not find them, a supplier is added under a slightly different name, or a product has a different code in each system. If you clean up the duplicates without changing the process that creates them, the number will be back where it started within about three months.
What we usually find
- The same customer recorded several times under different spellings
- Customer numbers that differ between systems, with no way to tell which is right
- Suppliers paid twice, through two records for the same company
- Products with different codes in sales, stock and finance
- A yearly clean-up exercise that never stays clean
- The same person contacted three times by the same campaign
Our position
Cleaning up duplicates without fixing the point where they are created is not a one-off project. It is a cost you will keep paying.
What the work covers
Match, merge and prevent
Matching records is the visible part of the work. Looking after the trusted record, and preventing duplicates when data is entered, are what make the results last.
Matching
We use exact rules where records share a common identifier, and likelihood-based matching where they do not. Both are tuned against your own records, so the balance between missed matches and false matches is a deliberate choice.
Choosing the best details
When records are merged, rules decide which source provides each detail. The trusted record takes the most reliable address, the latest contact details and the correct legal name.
The trusted record
There is one master record for each customer, product or supplier, with every source record linked to it. Each system keeps its own copy, but all of them agree on who is who.
Shared reference lists
Shared lists, such as countries, units and categories, are managed in one place and shared with every system, rather than kept separately and differently in each.
Preventing duplicates at entry
Matching runs when a record is being created. Before someone adds a new record, they are shown any likely existing match.
Ongoing ownership
Uncertain matches go into a review queue, owned by people who know the customers. Their decisions are fed back into the matching rules.
How it stays clean
Matching as an ongoing process
A trusted record is only as good as its latest update. These measures keep it current.
- Continuous matching
- New and changed records are matched as they arrive, rather than in a yearly clean-up. A duplicate is caught while there is still only one of it.
- Levels of certainty
- Records are only merged automatically when a match is almost certain. Less certain matches go to review, and every reviewer’s decision is recorded to improve the rules.
- Undoing merges
- Every merge can be reversed, with its history kept. A wrong merge does more harm than a missed one, so it must always be possible to undo it.
- Duplicate rate
- New duplicates are tracked by source and by where they were entered. This shows exactly where the process still needs to change.
- Shared identifiers
- The trusted record’s identifier is sent back to every source system, so reporting and operations use the same identifier to link records.
What you are left with
- One trusted record for each customer, product and supplier
- Agreed rules for which source provides each detail
- Duplicate checks when records are created
- A review queue owned by the business
- A duplicate rate tracked by source
Tell us how many customers you have
If the answer depends on which system you check, that difference is exactly where we start.