How we measure an AI code reviewer: 22 seeded pull requests, a scoring rule and every number

Auditor reviews pull requests in two layers. A rules pass checks the conventions a repository writes down, and a deep review looks for bugs: several passes with a frontier model propose findings, and a verifier model checks each one before it's posted. This article is about how we measure the second layer, and what the measurements say.

We wrote it because "our reviewer catches bugs" is a claim anyone can make. What matters is which bugs, how often, how much noise comes with them and what a review costs. Here is the method, the answer key's shape and every number from the runs we have, with the caveats that go with them.

Why seeded bugs

Measuring a reviewer on real pull requests sounds better, but it has a basic problem: nobody knows the full list of bugs in a real pull request. If the reviewer flags something, someone has to decide whether it's real. If it flags nothing, you can't tell a clean change from a miss. Recall, the share of bugs found, can't be computed without an answer key.

So each case is a pull request we built on purpose. It starts from a small, working repository (the base), applies a few exact edits (the head), and records which head lines hold the bug. A case that doesn't apply cleanly fails loudly, and expected lines are found by searching the head file for a piece of the planted code, so editing a case can't leave a stale line number behind.

Two choices keep the cases honest:

• The pull request body is empty. Each case has a description that names the bug, but it's only for the report. The reviewer sees the title and the code, nothing else.
• The titles sound harmless. The off-by-one case is titled "Support a page size of all remaining items". The missing-auth case is "Let users delete invoices". A reviewer that trusted the title would miss most of them.

The eval calls the same deep review code that reviews real pull requests, with the same prompts, models and voting. Only the source of the files differs.

The 22 cases

The first fourteen bug cases are small TypeScript repositories of two or three files, written for the eval. The two contract cases are the hardest kind for a reviewer that reads only the diff: the changed function looks fine on its own, and the code that breaks is in a file the pull request doesn't touch.

The documentation case gives the reviewer a documentation source, as a team would add one, and the only place the unit is written down is that source.

The three real-file cases are built from Shopify theme files, with eight bugs seeded into each, from easy to hard. They are client code, so we don't publish them. They also contain real bugs that were there before we seeded anything. A finding on one of those is correct but isn't a planted bug, so it's counted neither as a hit nor as noise.

Five cases are clean: a helper extracted from a loop, correct input validation, a harmless field added to a log line, a new pure function with a test, and one of the real theme files with only a variable renamed. Any finding on a clean case is a false positive, and we count those separately.

The scoring rule

A finding matches an expected bug when all of these hold:

1. It's in the same file.
2. Its line range is within three lines of the expected range.
3. Either one of the bug's keywords appears in its title, explanation or failure scenario, or its range overlaps the expected range exactly.

Keywords are deliberately loose. The SQL injection case accepts "injection", "sql", "interpolat", "parameter", "unsanitized" or "escape". The point is to check that the finding is about the bug, not to grade its wording.

Each expected bug matches at most one finding, and the best fits are paired first. A finding that names the bug beats one that merely overlaps its lines, so a broad finding can't take the slot of the specific one. Every finding left unmatched is a false positive, unless it is about one of a real file's existing bugs.

From that we compute precision (hits over all findings) and recall (hits over all planted bugs), plus how many clean cases got any finding at all.

Models, depths and cost

The eval runs both review depths with the platform's default models: anthropic/claude-sonnet-5.5 for the find passes and anthropic/claude-opus-5.5 for the verifier.

• Standard runs three find passes over the diff and the changed files, each reading the files in a different order. A finding at least two passes agree on goes to the verifier, as does any single High finding.
• Thorough runs four passes, each looking for one kind of problem: logic and edge cases, security, data and concurrency, and API contracts. Each can read, search and list the repository at the pull request's head. The verifier can explore too.

Cost is estimated from each call's token usage and the catalog price of the model's first certified provider. Time is per case, with four cases running at once.

Results on Oct 11, 2026

This is the most recent full run, all 22 cases at both depths.

By group:

The small cases are cheap: Standard spent $0.009 to $0.031 on each, Thorough $0.020 to $0.157. The real-file cases are larger and cost more, $0.13 to $0.15 each on Standard and $0.20 to $0.43 on Thorough, taking 22 to 44 seconds.

Thorough also made one finding on a bug that was already in one of the real files before we seeded it. It was right, and it's counted as neither a hit nor a false positive.

What the misses were

Standard missed four bugs on Oct 11.

Both contract cases. In contract-null-return, a lookup function now returns null instead of throwing, and a caller in another file still reads .email from the result. In contract-renamed-field, a serializer renames createdAt to created_at, and an untouched feed still sorts on createdAt. Standard reads only the files the pull request changed, and none of its three passes proposed anything on either case. On Thorough, all four passes can search the repository for callers, and each of them flagged both bugs.

logic-unhandled-null. A pull request titled "Drop the zero default from balanceOf" makes an unknown account id throw instead of returning 0. All three Standard passes found it. The verifier then rejected it, reasoning that the title said the default was dropped on purpose. Thorough's verifier opened the caller, saw that nothing guards against the throw, and confirmed it. Plausible titles are in the cases for exactly this reason.

One bug in a real-file case. Standard found seven of that file's eight; Thorough found all eight.

The one false positive was on data-nonidempotent-retry. Standard correctly flagged the dropped idempotency key, and also flagged that the new loop retries every error, including card declines, with no backoff. That's a fair point about the code, but it isn't in the answer key, so it counts against precision. We left the scoring as it is rather than widen the key after the fact.

Results over time

The first full run was on Oct 6, 2026, with the 17 cases we had then: no documentation case and no real-file cases.

The totals aren't directly comparable, since five cases were added in between. On the 17 cases both runs share, Standard found 10 of 13 both times, with one false positive each time. Thorough found 13 of 13 both times, and went from two false positives to none. On Oct 6 it flagged unvalidated input on the clean helper-extraction case, a real gap in the base code that the pull request didn't touch, and flagged a route in a logic case for returning a user's email. Between the runs the deep review learned to see removed lines, got better at placing findings, and gained platform knowledge for Shopify and documentation sources.

Standard's 10 of 13 hides a change: on Oct 6 it missed the dropped await and caught the null default, and on Oct 11 it was the other way round. Which is the next point.

Run-to-run variance

The same case doesn't always give the same answer. Model output varies between calls, and a bug that two of three passes find on one run may get one vote on the next.

The clearest sample we have is from Oct 7, when we ran one of the real-file cases several times in a row while tuning. Two Standard runs found all eight bugs, at $0.15 each. Three Thorough runs found seven, eight and seven, at $0.38 to $0.48, and the two runs that missed one each had one false positive. On that case, on that day, Standard did better.

So a difference of one or two bugs between depths, or between runs, is within noise at this sample size. The contract cases are different: Standard has no way to see the broken caller, and it found neither in either full run.

How we run it

The harness lives in our repository and calls the models directly. Its calls are platform-funded and never touch customer traffic.

pnpm auditor:eval                                     # every case, both depths
pnpm auditor:eval --depth standard --case contract-null-return
pnpm auditor:eval --find anthropic/claude-sonnet-5.5 --verify anthropic/claude-opus-5.5
pnpm auditor:eval --no-knowledge                      # without platform knowledge (A/B)
pnpm auditor:eval --no-docs                           # without documentation sources (A/B)
pnpm auditor:eval --dry-run                           # fake model: no network, no spend

--concurrency sets how many cases run at once (default 4), and --docs adds documentation sources from a JSON file to every case. Each run prints a table per depth and writes the scores, every finding and a trace of each pass to a JSON file per depth and day. The trace shows where a bug was lost: never proposed, outvoted, or rejected by the verifier. That's how we know the null-default miss above was the verifier's doing.

Before we change the deep review, we run the eval and compare against the last results, so a change that gains one bug and loses another shows up.

Limitations

These are the reasons not to read too much into the numbers above.

• The bugs are seeded, and the answer key is ours. We wrote the bugs, the keywords and the clean controls. A reviewer tuned against its own test set will look better on it than on code it has never seen.
• 22 cases is a small sample. One bug is 2.6 points of recall. The run-to-run swings above are the same size as the gaps we'd most like to measure.
• It covers the deep review only. The rules pass, static analysis, descriptions and fixes are not in these numbers. Real Thorough reviews also run the repository's type checker and linter first; the eval doesn't.
• The repositories are tiny. Most cases have two or three files, so Thorough's exploration has little to search. In a real repository the caller may be harder to find, and exploring costs more.
• Most of the code is TypeScript or theme code. Other languages aren't covered yet.
• Cost is an estimate. It's computed from token usage and catalog prices on one provider per model. A real review is billed per request, and Logs shows what each one cost.

We'll publish new runs as the case set grows, with the same tables, so the trend is visible and not only the best day.