← Back to all articles

Two AIs, one reviewing the other

Vladimir Dietrich · November 15, 2026 ·4 min read

For a few months now, I've been working with two artificial intelligences simultaneously. One, more expensive, thinks: it decides what to do, writes the instruction, and reviews the result. The other, cheaper, types: it edits the files, runs the tests, and returns a ten-line summary.

It works well. And it taught me something that applies outside of programming: summaries lie by omission.

The day when "green tests" meant nothing

In July, the cheap AI corrupted a file's syntax and returned a calm report: green tests. It was true. It had run a battery of tests that didn't even load the file it had edited itself.

Nobody lied. The report answered a slightly different question than the one I had asked, and I only read the answer.

A workbench with traps

So I stopped choosing the model based on what the manufacturer publishes and set up a small workbench, with two tasks designed to trick.

In the first, the agent needs to rename an option throughout the entire project. However, there's a key with the same name that's part of a communication protocol and cannot change. Renaming everything breaks the conversation with the other side.

In the second, it needs to replace many small queries with a batch query. There are five rules that the new version must respect, and no project test checks any of them. The one who checks is a judge the agent doesn't see.

Passing the tests, here, isn't enough. It was on purpose.

The green that hides

In September, one of the models renamed the option in the code and left two calls with the old name within its own tests. The tests remained green. The reason: when the option wasn't recognized, the program used a default value, and the default value happened to work.

I started calling this "the green that hides". The test passes, and it passes precisely because it stopped testing.

Anyone who works in a lab knows the relative of this: the experiment that "succeeded" because the control was wrong.

More effort didn't buy better results

Models have levels of effort: low, medium, high. I expected high effort to deliver more. On my workbench, it didn't.

In November, I measured the tokens for each round, which is what is actually spent. The low-effort model solved both tasks with about 112,000 tokens. The same model on medium effort spent 2.3 times that and delivered the same change. An "orchestrator" mode, which distributes steps to a helper, spent about 349,000 to get to the same place.

A month earlier, the generation change had been even clearer: 165,000 tokens versus 713,000, with the same result and less than half the steps. The old model didn't err. It took more detours.

The workbench is too easy to separate models into difficult tasks, and I know that. It answers only one question: for grunt work, which is the cheapest that doesn't fall into traps?

The workbench was also wrong

In the November round, I discovered that the test command for the tasks themselves had broken with a Node update. The tests didn't even run.

What saved the round was the agents' report. All sixteen said the command failed and showed which command they had used instead. No false greens. I redid the calculations with the correct command.

I also learned not to compare rounds from different days. The same model measured 165,000 tokens in October and 224,000 in November, with another tool version and another way of counting. I only compare numbers from the same round.

What remains with the reviewer

Once I let the cheap AI conduct five stages of a project on its own. Afterwards I reviewed: six serious defects, four of them already live, and the summary was wrong in all five stages. Reviewing afterwards cost the same as doing it directly.

With written instruction and review at each stage: no serious defects, and an accurate summary.

The serious defects didn't appear in the tests the agent wrote. They appeared when I ran the system against real data. The agent tests what it wrote. The error lies in what it assumed.

My rules today are few:

I read the change line by line, not the summary.

I ask for the actual command output, pasted, not "passed".

I keep a judge that the one doing the work doesn't see.

I measure the expenditure, not the impression of thoroughness.

I distrust my own ruler, because it also breaks.

The code and results of each round are open at github.com/vladimirbrasil/dupla.

Years ago I learned this from people: when the result is measured only by whoever produced it, everyone seems fine. With machines, it's being similar, for now.

In summary, tokens are yielding much, much more with this dual layer of different companies. I'll continue this way, for now.

Responses

Carregando respostas...

Deixe seu comentário

Respondendo a ✕