Our code reviewer has been wired to the wrong model for months. On a public benchmark, it barely mattered.
Our code has been sending AI code reviews to the wrong model since April. A public benchmark barely noticed. What got us from about 15 to 45.6 F1, and why we’re still 18 points behind Cubic.
In this note · 6 sections
Since April 15, our code has sent every Slopless pull request review to Claude Haiku 4.5, and it still does. Our architecture doc says Opus. I found the mismatch on August 7, while porting Martian’s open Code Review Benchmark (opens in a new tab) into our repo, and assumed it explained why our reviews weren’t better. It didn’t.
The benchmark has 50 real pull requests, a list of “golden” issues a good reviewer should catch on each, and scores for 21 tools. Cubic, the one we watched most, scores 63.6 F1. Our old reviewer scored 14.7 on Sonnet 4.5, and a replay of what production actually ran, the same reviewer on Haiku, scored 15.9. On the 34 pull requests both reviewed, each model found about 30 golden issues. The model barely mattered while the old code was in the way.
Over eight days in August I ran it 24 more times and got to 45.6. Those runs, plus the Haiku replay from November, are in experiments.csv.
The bug was one missing dictionary key
Slopless picks a model for each agent from a table. Trimmed:
_PRODUCTION_MODEL_TIERS: dict[str, ModelTier] = { "security-agent": ModelTier.OPUS, "feature-agent": ModelTier.OPUS, "polish-agent": ModelTier.SONNET,} def get_model_for_agent(agent_id: str) -> str: tiers = get_agent_model_tiers() tier = tiers.get(agent_id, ModelTier.HAIKU) return MODEL_TIER_MODELS[tier]The reviewer’s id is "pr-review-agent", and it isn’t in the table. Every review got the Haiku default, and nothing logged it. The fix on our development branch adds the entry and logs a warning whenever an agent falls back, because a missing entry in a table like this should be loud.
Measure the noise before you believe a result
Early on I ran one configuration twice with nothing changed. F1 moved by 3 points, and the second run found 4 fewer golden issues. When I repeated our final configuration at the end, F1 barely moved, but underneath it 33 of the roughly 100 issues either run found had flipped one way or the other.
So for the later changes I wrote down, before running them, how many golden issues each had to keep finding. Two prompt changes that raised F1 still failed that rule.
Three changes did most of the work
Each step in the order we took it. Precision, recall and latency for every step are in the CSV.
The first was the prompt. I rewrote it from a security scanner into a correctness reviewer, and F1 rose about 8 points. The same commit fixed a parse bug that made a crashed review look like a clean one, so part of that gain was the bug fix.
The second was deleting code. Our reviewer had grown to 2,125 lines: a finding taxonomy, a JSON schema the model had to fill in, a regex pre-scan and a confidence cap. I expected a bare prompt to do worse. The diff plus a short prompt scored about 5 points higher. It opens like this:
You are reviewing a pull request. Report defects a maintainer would fix before
merging: logic errors, broken API contracts, race conditions, data loss,
resource leaks, security holes, and documentation or tests that contradict the
code they describe.
The reviewer on our branch went from 2,125 lines to 550 the next day. GitHub needs structured findings, so a separate formatting call now turns the prose into a list, adding about 19 seconds on Opus.
The third was the model. The same short reviewer on Opus 4.8 instead of Sonnet 4.5 found 28 more golden issues, almost 9 points of F1, for about 6 seconds more per review.
Most of what I added lost real bugs
At the settings we tested, none found more golden issues than the reviewer without it. The architecture digest gained 2 on a single pass, inside the noise; shown is its three-pass retest. The shaded band is run-to-run noise.
File-reading tools were the idea I was surest of. I gave the reviewer read_file and list_files, and it really did go and read callers and types. Then it found 16 fewer golden issues. Doubling its step limit fixed most of the cut-off reviews, and it still found 13 fewer while taking 58 seconds instead of 51. A second call that re-judged each finding lost 12. Keeping only the old reviewer’s top three findings by severity dropped F1 from 14.7 to 10.0, because it had labeled its security-flavored guesses critical and described real bugs modestly. Several of these ideas raised F1 by cutting false positives while losing real bugs, which is why I stopped trusting F1 alone.
Two additions held up
One was a paragraph in the prompt: only report a defect whose failure you can show from the diff itself. On its own it lost 2 golden issues, inside the noise, and cut 42 false positives, six times the noise.
The other came from the noise itself. Two Opus runs with the same prompt each found 15 or 16 golden issues the other missed, so I ask three times and keep what at least two passes agree on. False positives fell by 32 and 39 in two runs while golden issues stayed within the noise. A review takes about 50 seconds before the formatting call.
Where we stand: 18 points behind Cubic
Most of the distance to the leaders is precision: 39% against their 59.5 to 67%.
Against Martian’s dashboard data, our 45.6 would rank 16th of 22 on F1. Cubic finds 104 golden issues and posts 65 false positives. We find about 87 and post about 135. To match its F1 without finding more, we’d need to get down to about 28 false positives and keep every real finding. I don’t think this architecture gets there.
Two limits on all of this: I tuned every change against the same 50 pull requests with no held-out set, and the judge is a Claude model scoring a reviewer built on Claude, though every tool on the dashboard faces the same judge.
All of this work, the tier fix included, is on a development branch that isn’t merged yet, so production still runs the old reviewer on Haiku. The engine was last deployed on August 3. Merging is worth about 30 points on this benchmark, and it’s the next thing we’ll do.
I ran these experiments and wrote this post. AI agents did much of the reading of misses and false positives. The benchmark, its golden comments and its scoring are Martian’s work. Ahmed Panju built much of the Slopless engine this reviewer runs inside, with contributions from Daniil Naumetc and Trevor Trinh.
