Built to work, or built to be run

Almost every system we are asked to rescue worked on the day it was handed over. It became a problem later, when the team changed, usage grew, or someone needed an answer the system was never designed to give.

There is a distinction that is usually lost during procurement. A system can be built simply to work, or it can be built to be run over the long term. The first is really a demonstration that happens to be live. The second is something a business can own. Both look the same at handover, but they cost different amounts, and only the second will still be serving the business in three years.

The distinction gets lost because nothing in a normal buying process tests for it. A demonstration shows what happens when everything goes right, and a specification lists features. Neither asks what happens at two in the morning when a queue backs up, or what a new engineer needs to read before they can safely change the pricing logic. Answering those questions costs money during the build but saves far more later. That makes them easy to put off, and expensive once they have been put off.

The three decisions that make the difference

In practice, the difference comes down to three design decisions made in the first release. None of them is unusual, but all of them are structural. Adding them later means rewriting parts of the system, not just adding to it.

Monitoring

In a system built only to work, errors come to light when a user reports them. In a system built to be run, the system produces structured logs, follows each request across services, and raises an alert about a problem before users notice it. The test is simple: when something breaks, can you tell what broke without having to reproduce it?

Deployment

In a system built only to work, a release is an event carried out by the one person who knows how. In a system built to be run, releases are routine, automated and reversible, and the same process is used for a one-line change and a major release. The test is whether anyone on the team can release on a Friday without a debate about whether they should.

Access design

In a system built only to work, everyone who needs access can get in. In a system built to be run, roles and permissions are designed, access is granted according to them, and removing someone’s access takes one action that is recorded. The test is how long it takes to answer the question “who can see customer data?”, and whether the answer is a list or an opinion.

Why the bill arrives late

The economics are what make this worth writing about. A system built only to work is cheaper to commission and costs about the same to run, until something happens that it was never designed for. Then the whole cost arrives at once, as an emergency, which is the most expensive way to pay for anything.

Imagine a business that needs to add a second warehouse. In a system built to be run, that is a configuration change. In a system built only to work, the location was assumed in the database design. Adding a warehouse then means migrating data, reworking permissions and spending a couple of weeks reconciling records. The requirement was reasonable, but the system had no way to accommodate it.

The same applies to people. A system built by one engineer who understood all of it is fine while that engineer stays. What makes a system easy to run is not documentation for its own sake. It is code, data and deployment that are structured and named clearly enough to show a new engineer what is going on. When documentation has to make up for an unclear system, nobody trusts it within six months.

What this looks like in the first release

• Structured logging and tracing included in the first version, not added after the first incident
• A deployment pipeline that runs from a code change to production without anyone doing steps by hand
• A role and permission model written down before the first user is created
• An audit trail on anything a customer or regulator might later ask about
• A database design that reflects what the business actually has, and allows for what it will have in future
• A guide to handling the three most likely failures, written by whoever would be called out

None of this makes up a large part of a build. On most engagements, it adds somewhere in the low tens of percent to the initial cost. It is also what allows the system to be extended rather than replaced. It tends to be cut because it cannot be seen in a demonstration, and demonstrations are what most buying decisions are based on.

When the cheaper option is right

Sometimes building a system only to work is the right choice, for example for a genuine prototype, something with a known end date, or a test of a business idea that will be thrown away either way. The mistake is not building the cheaper version. The mistake is building the cheaper version while believing you bought the other one, and only finding out when a customer asks for an audit trail that was never designed in.

So the question to ask anyone offering to build a system for you is not whether it will work. It will. The question is what happens when something changes, such as a new site, a new regulation, a new engineer or a bad afternoon. If the answer involves rewriting rather than reconfiguring, you are being offered the first kind of system at the price of the second.