An AI agent charges you per answer and never says whether the answer was any good

On one of the projects I work on we capture telemetry on every agent request.

It records the cost. It also records which prompt template was used, which model answered and how long it took.

That is more than a bill gives you. A bill tells you what the month came to. This tells you which design decision caused it.

We did not put it in at the start. It went in once we know the thing is going to scale and somebody is going to ask what it costs and whether it earned its keep. Before that there is nothing worth counting.

Even then it only measures the half that reports itself. Cost arrives on its own. Latency arrives on its own. Whether the answer was any good does not arrive in the telemetry report at all.

That half has to be built and the way we do it is not clever. The prompt templates carry the real policy constraints, the ones the business actually has, and we test against those properly. The test asks whether the output respected the constraint. There is no rubric to invent, because the policy was already written down by somebody whose job it was.

The useful part is that both land on the same unit. Cost per template from the telemetry. Quality per template from the policy tests. Same template, so you can read them next to each other.

That matters because the two move independently. Cost can go down while quality goes down with it and nothing red will appear anywhere.

There is a shape+build+launch idea that comes at this from the other end: the one metric that proves this works for real users. Telemetry will hand you every metric except that one.

Two longer write ups if this is where you are.

The cost is measured, the quality is not covers why quality has no meter and a couple of ways to build one.

A prompt template is a unit of measurement covers what goes into a template and why the name on it is what makes any of this possible.

What are you using to tell whether your output is any good?