AI enablement · Evaluation and model operations
What decides whether AI keeps working in production
We test the accuracy of your AI, re-test whenever a prompt or model changes, and manage model choice, caching, response times and costs. This operational work is what turns a successful pilot into something that lasts.
Where this starts
“It worked when we tried it” is not a measurement
AI systems fail differently from ordinary software. They do not show an error. They give an answer that is confident and wrong. They can also start doing this without anyone changing the code, because the provider updated the model or someone edited a prompt in a way that seemed harmless. Without measurement that runs all the time, quality depends on whatever the last person to try it remembers.
What we usually find
- Prompt changes released after trying just three examples
- An update to the provider’s model that quietly changed behaviour weeks ago
- No agreed definition of what a correct answer looks like for the task
- Rising AI costs, with no breakdown by feature or customer
- Response times that were fine in testing but not with many real users at once
- One expensive model used for every request, including the simplest ones
Our position
A prompt should be treated like code. It should be reviewed, tested and able to be rolled back. Otherwise, your product’s behaviour is being changed directly in production.
What the work covers
Measure quality first, then manage the cost
Quality comes first, because a cheap wrong answer is worthless. Cost and response time come next, because a correct answer that is too expensive to provide is not much better. Safety and injection testing is covered separately by AI security, and runs alongside this work.
Defining a correct answer
Before measuring anything, we agree with you what a good result looks like for this task, with worked examples. In most organisations, this conversation has never quite happened.
Test sets
We build a set of real cases with known good results, including the difficult ones, and keep it up to date. Results are marked automatically where possible, and by people where not.
Checks before release
Tests run automatically whenever a prompt or model changes. If quality falls below the agreed level, the release is blocked, just as it would be for a failing software test.
Choosing the right model
A cheaper, faster model handles the simple majority of requests, and the expensive one is used only where it is worth it. The split is based on measurement, not habit.
Caching and response times
We reuse answers where the same inputs come up again, and show results as they are generated where a wait cannot be avoided. The interface is designed around an agreed response time, rather than one discovered after launch.
Tracking costs
AI costs are broken down by feature, customer and environment. A prompt that suddenly becomes expensive is found within a day, rather than within a quarter.
How it stays measured
Quality that is reported automatically
All of this runs continuously, because AI behaviour can change even when nobody has touched the code.
Release check
Test results are attached to each build, with a pass mark. A drop in quality stops the release, rather than reaching customers and being reported back.
Model changes
We re-test regularly against a fixed set of cases. If the provider changes the model, you find out before anyone else does.
Sampling
A share of live requests is checked continuously. Real usage changes over time while a test set stays fixed, and surprises tend to appear in the gap between them.
Spending
The cost of each feature and each customer is tracked as it is incurred, with alerts on unusual spending, rather than being reconciled once a month.
Rollback
Prompts and model versions are fixed and can be reversed. A bad change is undone in minutes, rather than debated while it is still live.
What you are left with
- A written definition of a correct result for each task
- Up-to-date test sets, including the difficult cases
- A release check that blocks any drop in quality
- Model choices based on measurement, with the cost difference shown
- AI costs broken down by feature and by customer
Tell us how you currently know your AI is working
If the answer is that someone tries it now and then, that is the first thing to fix, and it is not expensive.