Data engineering · Pipelines and ingestion
Data that arrives every time, and on time
We build data pipelines that load only what has changed and run jobs in the right order. A failed run can be safely repeated, past data can be reloaded, and a change at the source is caught before it reaches a report.
Where this starts
Built for the days when everything works
It is easy to build a pipeline that works when everything behaves. The real engineering is for the other days. A source might send yesterday’s data twice, or a supplier might rename a field without warning. A job might fail halfway through and be impossible to re-run without duplicating a morning’s orders. Pipelines that are not built for these days fail without warning, and the first person to notice is usually someone reading a report.
What we usually find
- A job that creates duplicate records whenever it is re-run
- A renamed field at the source that quietly emptied a report
- Past data that cannot be reloaded without rebuilding everything
- Jobs scheduled by the clock, so one slow load breaks the next
- No record of what was loaded, when, or how many rows
- Everything reloaded every night, because loading only the changes was never built
Our position
If a pipeline cannot be safely re-run, someone will eventually have to fix it by hand, usually at the worst possible moment.
What the work covers
Moving data reliably into your data platform
This is different from connecting business applications, which moves records between systems to run a process. This work moves data into your data platform so that it can be analysed.
Loading only what has changed
Each run loads only what has changed since the last one. Loads stay fast and affordable as your data grows.
Safe re-runs
Every load is designed so that running it twice gives the same result as running it once. Most failures then become a simple retry rather than a clean-up job.
Reloading past data
Data for any past period can be reloaded without disturbing current data. When the logic is corrected, the correction can be applied to the past as well as the future.
Handling changes at the source
Changes to the structure of a source are detected and handled deliberately. A new or renamed field triggers a notification rather than an outage.
Running jobs in the right order
Each job runs when the jobs it depends on have finished, not at a fixed time that someone hoped would be late enough.
Speed matched to the decision
Data is updated daily where daily is enough, and almost instantly where a decision genuinely depends on it. Faster updates cost more, so they need to be worth it.
How problems stay small
Every run leaves a record
A pipeline’s value shows on its bad days. These measures keep those days short and visible.
Run log
Every run is recorded with its start and finish times and the number of rows read and written. A question about yesterday’s load can be answered in seconds.
Expected format
The expected format of each source is checked as it is loaded. If the check fails, the load stops rather than letting badly formed data through.
Retries
Temporary failures are retried after increasing intervals. Persistent failures go to the owner, with the failing step identified.
Late data
If a source has not arrived by its expected time, an alert is raised. Missing data does as much damage as wrong data, and it is harder to spot.
Cost per run
The computing cost and duration of each pipeline are tracked. A load that has quietly doubled in cost is investigated rather than simply paid for.
What you are left with
- Loads that only process changes and stay fast as data grows
- Pipelines that can be safely re-run
- The ability to reload any past period, using the same code
- Source changes caught when data is loaded, not in a report
- A run log showing what was loaded, when and how much
Tell us which report is most often late
The pipeline behind it is usually the right place to start, and the cause is rarely what people expect.