Answers from our docs and articlesesc to close
Try asking
Create an accountSign up

The bug our reviewer couldn't see: it never read removed lines

Auditor's deep review was shown only the new side of each diff, so a deleted guard or await was invisible. What we changed, and what the eval showed.

Key takeaways

  • Until Oct 7, 2026, Auditor's deep review saw only added and unchanged lines, so a bug made of a deletion, like a removed guard or a dropped await, had to be guessed from what was left.
  • On a case built from a real Shopify theme file with eight seeded bugs, three Thorough runs before the fix found six, five and five of them; two runs after it found all eight.
  • The same change stopped a rename-only control from drawing three findings about code the pull request never touched.
  • The numbers come from one seeded case and a few runs, scored against our own answer key, so they show direction rather than a measured rate.

For its first day, Auditor's deep review had a blind spot we didn't know about. It was never shown the lines a pull request removed. A deleted guard, a dropped await or a removed filter wasn't in its prompt at all. When it caught one of those, it was by inferring from the code that was left.

Here's how that happened, how our eval found it and what we changed. The eval itself is described in How we measure an AI code reviewer.

How the deep review lost the minus lines

Auditor reviews a pull request in two layers. The rules pass came first. It checks a repository's written conventions against the code a pull request adds, and comments on the new side of the diff. Its diff parser kept added and unchanged lines, numbered on the new side, and dropped every - line. For a rules pass that's reasonable: conventions apply to code that exists.

The deep review, which looks for bugs, was built later the same day on the same parser. Its prompt said so in as many words: report defects "introduced or exposed by the ADDED lines (marked +)".

Nothing failed. Line numbers were right and comments landed where they should. The diff the model saw was just missing half its content.

What a deletion looks like with no minus lines

Here's the kind of bug it hides. This isn't from the eval; it's a small example of the class. A refund function has two guards:

export async function refund(order: Order, amount: number) {
  if (order.status === "refunded") throw new Error("already refunded");
  if (amount > order.total) throw new Error("amount exceeds the order total");
  await payments.refund(order.chargeId, amount);
  await orders.markRefunded(order.id);
}

A pull request titled "Log refunds" adds a log line and, in passing, deletes the second guard. With removed lines dropped, the deep review saw this:

   11   export async function refund(order: Order, amount: number) {
   12     if (order.status === "refunded") throw new Error("already refunded");
   13 +   logger.info("refund", { orderId: order.id, amount });
   14     await payments.refund(order.chargeId, amount);

There's nothing wrong with any line shown. The log call is fine, and a function with one guard is a normal function. The only evidence of the bug is a line that isn't there. Now a refund larger than the order goes straight to the payment provider.

After the change, the same hunk reads:

   11   export async function refund(order: Order, amount: number) {
   12     if (order.status === "refunded") throw new Error("already refunded");
      -   if (amount > order.total) throw new Error("amount exceeds the order total");
   13 +   logger.info("refund", { orderId: order.id, amount });
   14     await payments.refund(order.chargeId, amount);

The removed line has no number, because it doesn't exist on the new side, where Auditor's comments go.

How the eval caught it

Not at first. Most early cases plant a bug by editing a line, and an edited line still shows as a + line, so the reviewer can often tell what was lost from what replaced it. What exposed it was a case built from a real Shopify theme file with eight seeded bugs, several of which remove or move code rather than change it. These are Thorough runs on that one case, on Oct 7, 2026:

RunsBugs found, per runFalse positives, per runCost per run
Before36, 5, 5 of 80, 1, 1$0.18 to $0.20
After28, 8 of 80, 0$0.21 to $0.22

The per-pass trace in each result made the cause clear. Before, the passes' proposed titles described the code as it now stood. After, many began "no longer…" or "moved…". The models could now say what had gone.

The before runs are scored with the rule we use now. The false positive in two of them was a correct finding placed too far from the bug's line to count.

The rename-only control

The same case has a clean twin: the same file with nothing changed but a variable's name. Any finding there is noise.

Before the change, a Thorough run on it posted three findings. All three were about problems that were already in the original file and that the rename didn't touch. Without minus lines, every renamed line looked like code written in this pull request, so the reviewer reviewed it as new. After the change, it could see each - line beside its + line, differing only in a name, and it posted nothing. The run also cost less, $0.090 against $0.137, because nothing went to the verifier.

That day they counted as three false positives. The case now lists the original file's existing issues, and a finding on one counts as neither a hit nor noise. Either way, comments on old code in a pull request that renames a variable are noise to whoever opened it.

What changed

It landed on Oct 7, 2026:

  • The deep review gets removed lines. Each is marked - and shown without a line number. The rules pass still reads only added lines, since conventions apply to code that exists.
  • The prompt says what a removal means. It now covers added and removed lines, and says "a removed call, guard, check, filter, await or cleanup is a bug when the remaining code still needs it; compare what was removed with what replaced it."
  • Findings on a removal have somewhere to go. The model is told to point at "the numbered line right where it was removed". A finding with no added line nearby now anchors to an unchanged line of the same hunk, where it used to be dropped. A hunk that only deletes is reviewed too.
  • Grouping stopped merging different bugs. One candidate covering a whole function used to absorb a separate bug inside it. Now overlap alone isn't enough; titles must also agree. Each pass may report up to 12 findings, up from 8, and must never combine two defects in one finding.

The eval gained the seeded case, the control, best-fit scoring and a per-stage trace in the same commit.

What it did to the small cases

The 14 small TypeScript cases include a dropped await. On Oct 6, before the change, Standard found nothing on it. On Oct 11, all three Standard passes proposed it, the first titled "Removed await on getUser breaks lookup", and the verifier confirmed it. The deep review changed in other ways between those runs, and results vary, so we don't credit all of it to this fix. But "Removed" in the title is the model citing a line it couldn't see before.

It isn't a cure. On Oct 11, Standard's passes all found another case's removed null default, and the verifier rejected it: it took the pull request's title as proof the removal was intended. That miss is covered in Standard vs Thorough.

The lesson

If you build a reviewer:

  • Check what the model actually receives. Print a real prompt and read it as the model would. We'd checked that comments landed on the right lines and never asked what the lines around them left out.
  • Seed bugs that are deletions. If every planted bug is an edited line, a reviewer that can't see removals will still score well. Remove a guard, an await, a cleanup call or a filter outright.
  • Pair seeded cases with clean twins. The rename-only control caught the opposite failure, a reviewer treating old code as new, which no bug case could show.
  • Keep a trace per stage. Knowing whether a bug was never proposed, outvoted or rejected by the verifier is what told us this was an input problem, not a model one.

The caveats are the usual ones and they matter here. The before and after numbers come from one seeded case and five runs, the bugs and the answer key are ours, and it measures the deep review only.

Frequently asked questions

Does Auditor comment on removed lines?

It reports bugs that are removals, but it places comments on the new side of the diff, so the comment goes on a line next to where the code was removed.

Does the rules pass see removed lines too?

No. The rules pass checks a repository's written conventions against the code a pull request adds. Only the deep review, which looks for bugs, is shown removed lines.

Did showing removed lines make reviews cost more?

Slightly, on the case we measured: $0.21 to $0.22 per Thorough run after the change against $0.18 to $0.20 before. On the rename-only control it cost less, because there was nothing left to verify.

One bill for your apps, your models and your reviews.

Host apps and functions next to your model traffic. Routing picks each turn's model by your quality floor, and Auditor reviews every pull request against your own rules.