Data engineering · Pipelines and ingestion
Data that arrives, every time, on time
Incremental loading, change data capture and orchestration, built so a failed run can be safely rerun, history can be backfilled, and a change at the source is caught before it reaches a report.
Where this starts
Built for the day it works
A pipeline is easy to build for the day everything behaves. The engineering is in the other days: the source that sends yesterday’s extract twice, the supplier who renames a field without notice, the job that fails halfway and cannot be rerun without duplicating a morning of orders. Pipelines built without those days in mind fail quietly, and the first person to notice is usually reading a report.
What we usually find
- A job that duplicates records whenever it is rerun
- A renamed source field that silently emptied a report
- History that cannot be reloaded without rebuilding everything
- Jobs chained by clock time, so one slow load breaks the next
- No record of what loaded, when, or how many rows
- Full nightly reloads because incremental loading was never built
Our position
A pipeline you cannot safely rerun is one you will eventually repair by hand, at the least convenient moment available.
What the work covers
Reliable movement into the platform
Distinct from integration between business applications, which moves records to run a process. This moves data into the analytical platform so it can be analysed.
Incremental loading
Only what changed since the last run, through change data capture or reliable change markers, so loads stay fast and affordable as volumes grow.
Safe reruns
Every load designed so running it twice gives the same result as running it once, which turns most failures into a routine retry instead of a cleanup.
Backfills
History reloaded for any period without disturbing current data, so a correction to logic can be applied to the past as well as the future.
Source change handling
Structural changes at the source detected and handled deliberately, so an added or renamed field becomes a notification rather than an outage.
Orchestration by dependency
Jobs run when what they depend on has finished, not at a clock time someone hoped would be late enough.
Latency matched to the decision
Daily where daily is enough, near real time where a decision genuinely depends on it. Faster costs more, and the difference ought to buy something.
How failures stay small
Every run leaves a record
The value of a pipeline is judged on its bad days. These keep the bad days short and visible.
Run log
Every run recorded with its start, finish, rows read and rows written, so a question about yesterday’s load is answered in seconds.
Contracts
The expected shape of each source checked at load, with a failed check halting the load rather than letting malformed data through.
Retries
Transient failures retried with backoff, and persistent ones escalated to the owner with the failing step identified.
Late arrival
A source that has not arrived by its expected time raises an alert, because missing data does as much damage as wrong data and is harder to spot.
Cost per run
Compute and duration tracked per pipeline, so a load that has quietly doubled in cost is investigated rather than absorbed.
What you are left holding
- Incremental loads that stay fast as data grows
- Pipelines that can be rerun safely
- Backfills for any period, through the same code
- Source changes caught at load rather than in a report
- A run log answering what loaded, when and how much
Tell us which report is most often late
The pipeline behind it is usually the right place to start, and the cause is rarely the one people expect.