Iterating on the decisions
Pull the decisions out of a document and you can see them, diff them and count them. Send them round a loop and they get sharper without covering more ground. Coverage is what decides how much gets invented.
A long piece of work is mostly decisions. What it covers, what it leaves out, which values it settles on, what nobody has answered yet. Normally those decisions only exist inside the finished thing, mixed into the prose, where you cannot see them, diff them or count them.
So pull them out. Hold them in a small structured object and write the document from that object rather than editing the document directly. That much is sound and it buys you things you did not have before. Two versions can be compared to see what actually changed. The whole state of the work fits on one screen. It is cheap enough to pass through a model many times.
The obvious next step is to refine the object in a loop until it stops changing, then write the document once at the end. That is the part these pages are about, because it is the part that does not work.
Say it another way
Imagine writing a plan by arguing about the plan itself, paragraph by paragraph. Every round you get a whole new plan and no way of telling which changes mattered. Now imagine keeping a short list of the actual decisions instead, arguing about the list and writing the plan out at the end. The second way is easier to check and easier to correct. The question these pages answer is what happens when you let a model do the arguing. The answer is that it gets more careful about decisions it has already made while staying blind to the ones nobody made at all.
What a refining loop can and cannot do
A refining loop has a fixed set of legal moves. Sharpen a decision, move one, remove one, close a question, open one. Every one of those rewrites or removes something already decided and none of them adds anything, so the object can only shrink. Precision is the only thing such a loop can improve.
That matters because precision is not what decides quality. Coverage is. How much a finished document has to invent depends on how many of its parts have a decision behind them. No amount of sharpening changes that count.
Measured against a raw arm, same input and same model with no machinery at all: eight decisions either way, seven parts covered either way, six parts invented either way.
Two things do pay. One is deciding in advance rather than iterating towards an answer. The other is closing the loop against the finished document rather than against the object.
Every number on these pages came out of a run rather than an impression. The example they were measured on is deliberately not the point.
The pages in this group
- The artifact is the only thing that knows what is missing. Precision is not coverage and coverage is what decides how much gets invented.
- Take the countable part off the model. Eleven times the answer was the same and it was never about asking more carefully.
- Checks that report green. The worst result a test can give you is a clean one it was never able to fail.
- Rules that pull against each other. Close every honest exit and a model will take a dishonest one.
- Everything you hand a model becomes material. Including your worked examples, your warnings and the diagnosis you wrote for yourself.
How to judge a method like this
This is the part that travels furthest, because it is not about loops. It is what to do when you have built machinery and you want to know whether it is doing anything.
- Measure the artifact, not the loop. The object converging is not the deliverable. It is possible to put twenty three commits of machinery in before anyone checks whether the document improved and to find that it has not.
- Always have a raw arm. Same input, same model, same final instruction, no machinery. If the machinery does not move the number against raw, it is not working, however good its internal reports look.
- Count something a substring cannot fake. If the measure can be satisfied by restating one decision five times, it will be and no guard can stop it, because it is a legal move.
- Rank every check by whether it needs the model to grade its own work. Prefer the ones that do not, say plainly which ones do and never let a self graded number be the headline.
- Run the failing question on a much larger model before blaming the model. The same failure at 8B and at 36B means the question is wrong. This one test saves whole rounds of pointless prompt work.
- Test both ends of every rule you add. A rule against inventing and a rule against stopping early pull in opposite directions and a fix aimed at one will land on the other.
- Establish that a change was in effect before crediting or blaming it. The most recent change is the most seductive wrong explanation available.
- Print the output and read it. A suite stays green through the bugs that are true of the code and false of the reader, which are the expensive ones. Both of the worst here were found by reading output.
- Drive it as a person, in the real environment, early. A headless check confirms what you told it to look at. Both times a person actually drove this one, they found a fault within minutes.
- Hunt for the check that cannot fail. A pattern that never matches, a rule with no site to fire at, a validator handed an empty list. All three report green.
The one paragraph version
Holding the decisions in a small object, separate from the document, is sound. It buys provenance, auditability and a convergence test that will catch a model claiming to be finished when it is not. Refining that object in a loop does not improve the document, because every legal move rewrites or shrinks what is already there and nothing adds. The loop only starts paying once it is closed against the finished document, which is the only thing that knows what is missing. Most of the value comes from deciding things in advance rather than from iterating. And when a question put to a model produces garbage, the answer is to find the countable part and take that part off the model, never to ask more carefully.