The cost is measured, the quality is not

Every call an agent makes has a price, and the price is easy to see. It lands on a bill, it is a number and it goes up. What the call was worth has no meter on it at all. So the one instrument you have points at the half that is easy to read, and the half that decides whether any of it was worth doing goes unwatched.

Two of the three arrive on their own. The one that decides whether any of it was worth doing has to be built, and there are a few ways to build it.

The state to watch for

Here is a change that reads as a clear win. A set of rules moves out of the model and into code. The tests stay green. The cost goes down. The results get worse.

Every instrument reports a success. Nothing in the pipeline is in a position to notice otherwise, because nothing in it knows what the work was for.

That state does not look like failure from the outside:

• cost down
• tests passing
• output worse

If you are watching the first two, it reads as a win.

Why the tests do not catch it

A test asserts that a specific thing happened. That works where there is one right answer. Where the work needs judgement, a test can check the shape of the answer and not whether it was the right judgement.

So you can hold full coverage over a pipeline that has started producing worse work, and nothing goes red, because nothing was ever asked to hold an opinion.

Every request is a subscription

If every request needs an agent to resolve it, you have not bought a feature. You have signed up to a subscription for that service. That is fine when what comes back is worth it.

It changes the question though. Not what the month cost. What each call bought.

The order that works

Measure the quality of what is being done first. The cost decisions come after that, in how you design the pipeline.

Doing it the other way round is what produces cost down, tests green and results worse. You optimise the half you can see, because it is the half you can see.

A couple of ways to capture it

None of these is a product you can buy. They are all small, and they differ in what they cost you and what they catch.

A set of cases you judged by hand. Ten or twenty real inputs with the answer you would have given, written down before you start and then kept fixed. Costs an afternoon and goes stale. Catches regressions on the work you already know matters. Without this one, none of the others have anything to be right about.

A model scoring them against written criteria. Once you have your own verdicts on the fixed set, a model can score everything else the same way. Costs tokens on every run and needs recalibrating against your verdicts or it drifts. Catches volume, which is the part nobody has time to look at.

Reading the output. No instrument at all. Somebody who knows what the thing is for, looking at what came out. Costs your time and does not scale. Catches what nobody thought to write a case for. It is also the only one that finds a problem you had not imagined.

Whichever you pick, put the number next to the cost number and read them together. They move independently, so either one on its own will tell you a comfortable half of the story.

None of this needs a platform. All three depend on somebody holding what the work is for, because without that a drop in quality has nothing to be measured against. That sounds soft and it is the most reliable instrument on this page. It is also the thing nobody writes down.