Between 03:54 and 04:57 this morning I was trying to get one LinkedIn carousel past a critic. The carousel is Jev v2, about a model I had tested inside my browser agent. The critic is another model call. It reads the pages, the caption and the facts file, and returns a score out of 10 plus a list of what is wrong. My rule was simple: fix what it names, run it again, ship when it passes.
That rule did not survive the morning. Here is why, from the commit log.
Five rounds of one critic
Round 1, 03:54. 3/10, Truth fail. Fair: an invented score distribution on page 4, a phrase in no source file, and "fast, cheap" on the last page when page 9 said otherwise. I rewrote seven pages and the caption.
Round 2, 04:03. 3/10, Truth fail again. The fix this time was one denominator for every figure, "of 47". I also cut a row on page 9 that compared two different tests as if they were one.
Round 3. 4/10, no Truth fail. Progress. The critic listed story problems, and I fixed each one in round 4. Page 1 now named the two tests. Page 3 was cut to one bar. Page 8's checklist stopped being tied to one tool. Page 10 closed on the caption's question.
Round 5, 04:36. 3/10, Truth weak.
Every item round 3 named had been fixed, and the score went down. The critic had not become stricter. It had found a different top problem: whether the carousel said Jev chose a post or judged one it was handed. That was a real point. But round 3 had not raised it, and the round after 5 might have raised something else again.
What the drop actually meant
A critic's verdict is one sample. Ask the same model the same question twice and you get two different ranked lists, and the top of each list decides the score. Fix the first list and you have not fixed the second, because you never saw it.
So "fix what it names and rerun" is not a loop that converges. Each round removes what one sample happened to rank first, and the next sample puts a new problem on top. I had read the 3 after the 4 as the carousel getting worse, but the round 5 pages carried every fix round 3 had asked for. The instrument was noisy, and I was treating each reading as the truth.
The gate I replaced it with
At 04:40 the gate changed, in code rather than in a prompt:
- Three critics, each fresh. Each runs in its own context and gets only file paths: never the prompt chain, never the other critics, never my reasoning about what I fixed. The isolation is the point. A critic that can see the last verdict just argues with it.
- The median score decides, not any single score.
- An item counts only when two of the three name it. A problem one critic raises goes on the list as a note, not a blocker. Two of three is a pattern.
- One Truth fail with evidence still blocks on its own. Taking a median is fine for taste. It is not fine for a false claim. If one critic can point to the sentence that says something the sources do not, one vote is enough.
Round 6 took on the judge-versus-choose point from round 5, because it was a truth issue and it was right. The copy now says the model judges a post it is handed and cannot choose one. The structural check passed 13 of 13.
Then the new gate ran: 5, 5, 5. Median 5. No Truth fail. Five items were named by two of three critics: a short list I could trust.
At 10:13 the carousel went into the queue for Thursday 24 September, 09:00 IST. Median 5 is my own call and not a high bar; the plan is to raise it slowly, not to pretend a 5 is a 9.
It held up on the second carousel
Later in the day the same gate passed a second carousel, lint-noise, in round 2
at 5, 5 and 6. The first real run of the carousel phase did find three defects,
all in the plumbing and none in the critics. The one worth repeating: critics
that can only read files cannot run git, so a fact that lives only in git now
goes into a facts file as raw git show output, never retyped.
What I take from it
If you use a model as a reviewer, a single run of it cannot tell you whether your last fix helped. What it gives you is one draw from a distribution. Take several independent draws, and keep them independent. Count what shows up repeatedly. Let the median decide questions of quality. And keep one exception that a vote cannot override: a claim that is not true.
The score fell from 4 to 3 on better pages, and that was the useful result of the morning. It showed me the thing I was measuring with, not the thing I was measuring.
