Answers from our docs and articlesesc to close
Try asking
Create an accountSign up

Every reviewer eval needs pull requests with nothing wrong

Precision on seeded bugs doesn't show whether a reviewer can stay quiet. How we build clean controls for Auditor's eval, score them and what they showed.

Key takeaways

  • A seeded-bug eval only asks a reviewer about pull requests that contain a bug, so it can't show what the reviewer does with a change that's fine.
  • Auditor's eval has five clean pull requests: four small TypeScript changes and one built from a real Shopify theme file with only a variable renamed.
  • Any finding on a clean case counts as a false positive, and we also count how many clean cases drew any finding at all.
  • On Oct 11, 2026, neither review depth flagged any of the five, but five cases is a small sample and the controls are ours.

Plenty of pull requests have no bug in them. A reviewer that comments on them anyway teaches people to skip its comments, real findings included. A seeded-bug eval can't measure that on its own. The rest of the method is in How we measure an AI code reviewer.

Why precision on seeded bugs isn't enough

Precision does count noise on a bug case: finding the bug and three other things scores 25%. But every bug case has something to find, so a reviewer that always reports at least one finding is rewarded there. Precision on seeded cases measures the noise that comes alongside a real bug. It can't show whether the reviewer can look at a correct change and say nothing.

So the eval has cases where the right answer is an empty list, and the reviewer's prompt says to return {"findings":[]} when it finds nothing.

How the controls are built

A control is built like a bug case, from a small working repository and a few exact edits, with nothing planted. Four reuse a bug case's small TypeScript repository, so the reviewer reads the same files it reads to catch the bug next door:

  • Extract the page bound into a helper. The loop bound in a pagination function, Math.min(items.length, offset + size), moves into a named pageEnd helper. Behavior doesn't change. The off-by-one case breaks the same line.
  • Reject non-numeric offsets. The orders route checks that offset is a non-negative integer and returns a 400 otherwise.
  • Log charge amounts in dollars. A log line gains dollars: (cents / 100).toFixed(2). The secret-logging case leaks a key into the same line.
  • Add a slugify helper. A new pure function and a test, touching nothing else.

The fifth is built from a real Shopify theme file, with only a variable renamed. It's client code, so we don't publish it. Being real, it has problems of its own that predate the rename, which makes it a harder test: a reviewer can be right about the code and still wrong to comment on this pull request.

How they're scored

Every finding on a clean case is a false positive. The run reports the total, and separately how many controls drew any finding at all. The second matters more: three findings on one control and one each on three controls are the same total, but the second reviewer comments on most correct changes.

The real-file control carries a list of the issues already in the original file, matched by file and keyword. A finding on one of those is right about the code, so it counts as neither a hit nor noise. The small controls have no such list, so a true point about their unchanged code still counts against the reviewer.

What they've shown

RunDepthClean casesFlaggedFindings on them
Oct 6, 2026Standard400
Oct 6, 2026Thorough411
Oct 11, 2026Standard500
Oct 11, 2026Thorough500

The one flag on Oct 6 is the useful one. Thorough, reviewing the helper extraction, reported that a client-controlled offset isn't validated, so a negative or fractional value pushes undefined items. That's true of the base code, and it's exactly what the validation control adds, but this pull request didn't touch it. Right about the repository, wrong about the change: to whoever opened the pull request, that's noise.

The real-file control was added on Oct 7. A Thorough run on it that day drew three findings, all about problems already in the original file: the deep review was then shown only added lines, so every renamed line looked new. The bug our reviewer couldn't see covers that fix. Since it, the control has drawn none.

Clean cases are cheap to run. On Oct 11 the small controls cost $0.009 to $0.020 each on Standard and $0.020 to $0.048 on Thorough, since with nothing proposed, nothing goes to the verifier.

The caveats

Five clean cases is a very small sample: zero flags out of five is consistent with a reviewer that comments on one correct pull request in ten. We wrote the controls, and four are a few lines in a two- or three-file repository. Large correct changes are harder to stay quiet on, and we need more controls like the real-file one. The eval also covers the deep review only, not the rules pass or static analysis.

If you build a reviewer eval, add as many cases with nothing wrong as you can, on the same code as the bug cases, and report how many drew any comment, not only the total.

Frequently asked questions

What is a clean control in a code review eval?

A pull request with no bug in it, where the correct output is no findings. Any finding the reviewer makes on it is counted as a false positive.

What if the reviewer finds a real problem in a clean case?

On the small controls it still counts as a false positive, because the pull request didn't cause it. The control built from a real theme file lists that file's existing issues, and a finding on one of those counts as neither a hit nor noise.

Does a clean review cost anything?

Yes, but little. On Oct 11, 2026 a small clean change cost about one to five cents in our eval, depending on depth. With nothing proposed, nothing goes to the verifier.

One bill for your apps, your models and your reviews.

Host apps and functions next to your model traffic. Routing picks each turn's model by your quality floor, and Auditor reviews every pull request against your own rules.