AI enablement · Evaluation and model operations
The part that decides whether it survives production
Task accuracy evaluation, regression when a prompt or a model changes, routing, caching, latency and token cost. The operational layer that separates a durable capability from a successful pilot.
Where this starts
It worked when we tried it is not a measurement
AI systems fail differently from ordinary software. They do not throw an error; they return something confident and wrong. And they can begin doing so without anybody touching the code, because a provider changed a model underneath you or because a prompt was edited in a way that read as harmless. Without a measurement that runs continuously, quality becomes whatever the last person to try it happens to remember.
What we usually find
- Prompt changes shipped on the strength of trying three examples
- A provider model update that quietly changed behaviour weeks ago
- No agreed definition of what a correct answer even is for the task
- Token spend rising with no attribution to feature or customer
- Latency that was acceptable in testing and is not at real concurrency
- One expensive model used for every call, including the trivial ones
Our position
A prompt is code. It ships through review, it has tests and it can be rolled back. Anything less means the behaviour of your product is being edited directly in production.
What the work covers
Measurement first, then economics
Quality before cost, because a cheap wrong answer is worth nothing. Then cost and latency, because a correct answer nobody can afford to serve is only marginally better. Safety and injection evaluation belongs to AI security and runs alongside this rather than inside it.
Defining correct
Agreeing what a good output actually is for this task, with worked examples, before anything is measured. In most organisations this is the conversation that has never quite been had.
Evaluation sets
Real cases with known good outputs, including the awkward ones, maintained as an asset rather than assembled once. Graded automatically where the task allows and by people where it does not.
Regression gates
Evaluations running in the pipeline on every prompt and model change, with a threshold that blocks a release exactly the way a failing test does.
Model routing
The cheap fast model for the easy majority and the expensive one only where it earns its place, with the split decided by measurement rather than by preference or habit.
Caching and latency
Caching where inputs repeat, streaming where a wait is unavoidable, and a latency budget the interface was designed around rather than discovered by after launch.
Cost attribution
Token spend broken down by feature, customer and environment, which is how a runaway prompt is found within a day instead of within a quarter.
How it stays measured
Quality that reports itself
All of this runs continuously, because the thing being measured can change without anybody touching the code.
Gate
Evaluations attached to the build with a passing threshold, so a quality regression stops a release rather than reaching customers and being reported back.
Drift
Scheduled re-evaluation against the fixed set, which is how a provider-side model change becomes visible to you before it becomes visible to anyone else.
Sampling
A share of live traffic graded continuously, because an evaluation set ages while production does not, and the gap between them is where surprises live.
Spend
Cost per feature and per customer tracked at the point the tokens are spent, with alerting on anomalies rather than a monthly reconciliation.
Rollback
Prompts and model versions pinned and reversible, so a bad change is undone in minutes rather than debated while it is still live.
What you are left holding
- A written definition of a correct output, per task
- Maintained evaluation sets, including the awkward cases
- A release gate that blocks on quality regression
- Model routing chosen by measurement, with the cost difference shown
- Token spend attributed by feature and by customer
Tell us how you currently know it is working
If the answer is that somebody tries it occasionally, that is the first thing to fix and it is not expensive.