Software engineering · Reliability and operations
Running a system is most of the job
We set service level objectives and put in place monitoring, incident response and cost control, for systems we have built and systems built by others. Running what we build is part of our standard, and this is how we make sure it works.
Where this starts
Nobody owns production until something breaks
Responsibility for running a system is often never formally agreed. That works until the first serious incident, and then it does not. Alerts go off and nobody acts on them. The people who could fix the problem are not the ones who are told about it, and the review afterwards focuses on who was to blame rather than what went wrong. Meanwhile, cloud costs rise steadily, because no single change is ever big enough to question.
What we usually find
- Alerts that go off regularly and are regularly ignored
- No agreed definition of what “available” means for each service
- Incidents handled by whoever happens to notice first
- Logs that exist, but cannot answer questions during an outage
- Rising cloud costs, with no breakdown by service or team
- Performance problems reported by customers before the monitoring detects them
Our position
If an alert does not need action, it should not wake anyone up. Every ignored alert makes it less likely that the next real one will be answered.
What the work covers
Define it, monitor it and take responsibility for it
Reliability starts as an agreement about what the business needs, and only then becomes a technical exercise. The engineering follows from that agreement, not the other way round.
Service level objectives
A clear, measured definition of acceptable service, agreed with the business rather than chosen by engineers. It includes an error budget, which makes the trade-off between reliability and delivery speed explicit.
Monitoring
Logs, metrics and traces that let someone who did not build a service diagnose it in the middle of the night. They are structured and linked together, rather than simply large in volume.
Alerting
Alerts are based on the agreed service levels rather than raw thresholds. They go to people who can act on them, and there are few enough that people continue to take them seriously.
Incident response
A defined process and a severity scale that the team has practised. Reviews after an incident focus on what went wrong rather than who, and produce actions with owners and dates rather than a document that gets filed.
Capacity and performance
We test the system under realistic load, find the bottleneck and plan how it will grow before the busy period arrives, not during it.
Cost control
We break down spending by service, team and environment, and identify the changes that really matter. A small number of decisions usually account for most of the bill.
How it keeps working
Anything that relies on memory will eventually be forgotten
Everything below runs whether or not anyone is watching. It is the only approach that survives holidays, resignations and busy periods.
- Service levels
- Service levels are measured continuously, and both engineering and the business can see how much of the error budget has been used. The conversation happens before a target is missed, not after.
- Health checks
- Checks test the real services a system depends on, not just whether it is switched on. A service can be running perfectly and still be useless.
- Recovery
- Failover and restores are automated and rehearsed on a schedule. The recovery time is measured, rather than quoted from the last time it happened.
- Capacity
- The system scales based on the measure that best predicts your load. Limits are set, so a process that gets out of control cannot also create an out-of-control bill.
- Cost
- Unusual spending and unused resources are detected automatically and reported to the team that owns them, rather than waiting for a monthly finance review.
What you are left with
- Service levels agreed with the business and measured continuously
- Dashboards and traces that answer questions during an incident
- An on-call rota and incident process that your team can run
- A rehearsed recovery process, with a measured recovery time
- Cloud spending broken down by service, and the decisions that affect it
Tell us what an hour of downtime costs your business
We will work out what it takes to prevent it, and what it costs to keep it that way.