Software engineering · Reliability and operations
Running it is the longer half of the job
Service level objectives, observability, incident response and cost control, for systems we built and systems we did not. This is the part of running what we build that has to be demonstrably true.
Where this starts
Nobody owns production until something breaks
Operational responsibility is usually left implicit, which works until the first serious incident and then does not. Alerts fire that nobody acts on, the people who can fix a problem are not the people told about it, and the review afterwards concentrates on who rather than what. Meanwhile cloud spend rises steadily, because no individual change is ever large enough to be worth questioning.
What we usually find
- Alerts that fire routinely and are routinely ignored
- No agreed definition of what available means for a given service
- Incidents handled by whoever happens to notice first
- Logs that exist but cannot answer a question during an outage
- Cloud spend rising with no attribution to service or team
- Performance problems reported by customers before monitoring sees them
Our position
An alert that does not require action should not wake anyone. Every page that gets ignored makes the next real one less likely to be answered.
What the work covers
Define it, watch it, answer for it
Reliability is an agreement about what the business needs before it is a technical exercise. The engineering follows from the agreement rather than the other way round.
Service level objectives
A stated, measured definition of acceptable, agreed with the business rather than chosen by engineering, with an error budget that makes the trade-off against delivery speed explicit instead of implicit.
Observability
Logs, metrics and traces that let somebody who did not build a service diagnose it at three in the morning. Structured and correlated, rather than merely present and voluminous.
Alerting
Alerts tied to the objectives rather than to raw thresholds, routed to people who can act on them, at a volume low enough that they stay credible.
Incident response
A defined process, a rehearsed severity scale, and a blameless review that produces owned actions with dates rather than a document that is filed.
Capacity and performance
Load testing against realistic volume, with the bottleneck identified and the scaling path known before the peak arrives rather than during it.
Cost engineering
Spend attributed by service, team and environment, with the changes that actually matter identified. A small number of decisions usually account for most of the bill.
How it holds without vigilance
Anything that needs remembering will be forgotten
Everything below runs whether or not somebody is watching, which is the only arrangement that survives a holiday, a resignation or a busy quarter.
- Objectives
- Service levels measured continuously with error budget burn visible to engineering and the business, so the conversation happens before the breach rather than after it.
- Health
- Deep checks that exercise real dependencies rather than confirming a process is alive, because a service can be running perfectly and still be useless.
- Recovery
- Automated failover and restore, rehearsed on a schedule, with the recovery time recorded as a measurement rather than quoted from the last occasion.
- Capacity
- Scaling driven by the signal that actually predicts load for your workload, with limits in place so a runaway process cannot become a runaway bill.
- Cost
- Anomaly detection on spend and idle resource reporting, routed to the team that owns the account rather than to a monthly finance review.
What you are left holding
- Objectives agreed with the business and measured continuously
- Dashboards and traces that answer questions during an incident
- An on-call rotation and incident process your team can run
- A rehearsed recovery with a measured restore time
- Cloud spend attributed by service, and the decisions that move it
Tell us what a bad hour costs your business
We will work out what has to be true to prevent it, and what it costs to keep that true.