AI Code Review Bottleneck: What CMU and Stanford Measured

The number that stopped me was not the 2.09x throughput. It was 39% to 21%. In the study I am about to walk through, the share of pull requests that received any written comment from a human reviewer nearly halved in under two years, while the merge rate and revert rate stayed flat. Every dashboard a team would normally watch stayed green while the thing review exists to do thinned out underneath it.

This post reads the paper itself, not the summaries: Hao He, Shyam Agarwal, Yegor Denisov-Blanch, Pavel Azaletskiy, Sanmi Koyejo and Bogdan Vasilescu, arXiv 2607.01904, submitted 2 July 2026. I pulled the PDF and read the methods and threats-to-validity sections, then compared it with how Semgrep's 9 October 2026 essay used it. It is a different angle from my earlier posts on AI review as a CI gate and on the DORA throughput and instability finding: this one is about reviewer capacity, what the data can and cannot prove, and what to set up before your own queue fills.

What the study actually measured, and on whom

It is a longitudinal case study of one mid-sized B2B software company with an R&D organisation of several hundred engineers. In June 2025 its CTO announced a 2x mandate: double engineering productivity through AI adoption over twelve months, with merged pull requests per engineer per month as the progress metric. The company gave the researchers access to its PR history and its Cursor and Claude Code usage logs. The authors say the company did not take part in the analysis or writing and no employee is an author.

The panel covers 802 developers and 196,212 non-bot pull requests across 364 active repositories, January 2024 to April 2026. 30.2% of those PRs carry the company's own created-by-ai label, and the authors treat any AI-versus-human PR comparison as correlational because the label was not assigned by them. The causal claims rest instead on a staggered difference-in-differences design that compares each developer with their own earlier output.

Two choices in the data matter for everything that follows. Bot reviews are separated out: automated accounts made up roughly 38% of review rows, so the authors count the first non-author, non-bot review as the start of the review phase. And the company is deliberately unrepresentative. The paper calls it young, AI-forward and permissive, funding AI tools without per-seat caps or token budgets, and says its effects are better read as an upper envelope than an industry average.

The numbers, with the paper's own definitions

Here is every figure I use, taken from the paper's text and tables rather than from a secondary write-up. The right-hand column matters as much as the middle one, because several of these numbers are easy to misquote when the definition is dropped.

Notice what is not in the table: a measured defect count. The paper has none. That absence is the subject of section five.

Silent approvals: where the extra load actually landed

The mechanism is plain arithmetic. Volume grew 3.1x and the pool of people reviewing grew 1.5x, so each reviewer carries about twice the work. The organisation did not hire its way out and did not let a queue grow unbounded. It did two things: it routed review to automation, and the human review that remained got thinner.

The thinning is measurable at two levels. Per PR, written feedback fell from about 39% to about 21%. Per reviewer, the median person still left about three commented reviews a month, the same as before, while their approve-only reviews roughly doubled. In the authors' words, the added load fell on bare approval rather than substantive review.

Be careful with one phrase from the Semgrep essay. It says silent approvals per reviewer doubled, which matches the paper's per-reviewer median. But as a share of PRs, silent approvals stayed about flat near 50%. Both are true; they answer different questions. The first says a person is clicking approve twice as often. The second says the proportion of PRs that get only a click did not change much, because the proportion that got no human at all grew.

The 20% latency: read it as a queue signal, not a quality signal

At the PR level the authors compare AI-authored and human PRs within the same author and calendar month, controlling for change size. AI PRs took about 20% longer from first human review to merge post-mandate, and about 22% longer in total cycle time. Pickup, the wait before a human first looks, was also longer in the paper's Table IV, and the review phase was longer still.

The paper's reading is that this is a queue, not rework. Review rounds and change-request counts are flat, and AI PRs are no less likely to get a human review. What lengthens is the interval between first review and merge, over which AI PRs draw about 9% more comments without more revision rounds. The authors call it more reviewer engagement, not more fixing.

Two limits from the paper keep this honest. The estimate is correlational because the created-by-ai label may track unmeasured complexity. And late in the window the company began routing PRs through AI-driven review and auto-approval, which collapses human latency for a growing share of PRs. The authors say that biases the measured premium toward zero, so 20% is conservative for the regime where humans still review.

What the data cannot tell you: green dashboards, no defect count

The headline reassurance is that merge rate stayed flat and revert rate drifted down, so the extra volume shipped rather than piled up. The authors then undercut it themselves. Merge and revert rates are coarse, short-horizon proxies that miss defects, incidents and maintainability, and they say to read the result as evidence against an acute quality collapse, not as a clean bill of health. The company also deploys every PR automatically, so a defect that does not trigger a revert is invisible to both metrics.

That is the named failure in this story: a review process can degrade while every metric the team watches stays flat, because the metrics were chosen for a human-paced pipeline. The paper's own discussion says DORA and SPACE register the speedup but miss where the work went. If your merge count is the target, as it was here, the pressure is to keep the count rising while the reviewer-side cost goes unmeasured.

Two smaller cautions. The adoption-and-use link to throughput is read as an association because developers chose how heavily to use AI, so the paper does not claim an exact causal size. And I left out one figure that Semgrep's essay attributes to a 2022 ICSE review study, a human catch rate of 3% to 61%. I read that paper's abstract and it reports 150 participants and an eightfold detection gain from asking reviewers to focus on security; I could not confirm the 3% to 61% range from what I read, so it is not used here.

Semgrep's exception-only model, and what it still has to prove

Semgrep's essay, written by Daghan Altas and dated 9 October 2026, proposes flipping the default: every change is reviewed by agents for correctness, security and architectural alignment, routine changes proceed when the evidence meets policy, and a human is pulled in only for consequential ones such as new service boundaries, major dependencies or threat-model changes. Its stated steady-state target is fewer than 1% of PRs needing a human. Semgrep sells security tooling, so read this as a vendor's proposal, not a result.

The part I find credible is the rollout, not the 1% target. Semgrep's pilot runs agents in shadow mode next to human reviewers, tracks false approvals, coverage, cost and speed, and requires four checks to all pass: knowledge transfer, product, design and correctness. Failed checks are retried by the agent, and after repeated failures a human takes over. Experts label real PRs as test cases first. That is measurement before trust.

Anthropic's AI-native SDLC playbook points the same way but keeps a human gate: it describes layers of agentic review with human review reserved for regulated and critical code, says findings do not approve or block a PR on their own, and says branch protection still requires approval from a code owner. Separation of duties is preserved because the agent that wrote the code cannot approve it. Notice the design difference: Semgrep's target removes the human from most PRs, while the playbook keeps a code-owner approval as the final control.

Measure your own review load before you copy anyone's fix

The study's metrics are reproducible on a smaller scale from GitHub's GraphQL API. This script takes the last N merged PRs of a repository and prints the share with any written human feedback, silent approvals per reviewer, and the median hours from first human review to merge. I wrote it to mirror the paper's definitions: a human is a non-author, non-bot user, and written feedback is a non-empty review body or at least one inline comment. I have not run it against a live repository from this environment, so test it on a small repo first.

// Usage: node review-load.mjs OWNER REPO [N]
// Measures, for the last N merged PRs: share with any written human feedback,
// silent approvals per reviewer, and hours from first human review to merge.
import { execFileSync } from "node:child_process";

const [owner, repo, n = "100"] = process.argv.slice(2);
const query = `
query($owner: String!, $repo: String!, $n: Int!) {
  repository(owner: $owner, name: $repo) {
    pullRequests(states: MERGED, last: $n, orderBy: {field: UPDATED_AT, direction: ASC}) {
      nodes {
        mergedAt
        author { login __typename }
        reviews(first: 50) {
          nodes { state body submittedAt author { login __typename } comments { totalCount } }
        }
      }
    }
  }
}`;
const raw = execFileSync("gh", ["api", "graphql", "-f", `query=${query}`,
  "-F", `owner=${owner}`, "-F", `repo=${repo}`, "-F", `n=${n}`], { encoding: "utf8" });
const prs = JSON.parse(raw).data.repository.pullRequests.nodes;

let withFeedback = 0, reviewed = 0;
const silentBy = new Map();
const hours = [];
for (const pr of prs) {
  const human = pr.reviews.nodes.filter(
    (r) => r.author && r.author.__typename === "User" && r.author.login !== pr.author?.login);
  if (human.length === 0) continue;
  reviewed++;
  // "Written feedback" = a review body or at least one inline comment.
  if (human.some((r) => r.body.trim() !== "" || r.comments.totalCount > 0)) withFeedback++;
  for (const r of human) {
    const silent = r.state === "APPROVED" && r.body.trim() === "" && r.comments.totalCount === 0;
    if (silent) silentBy.set(r.author.login, (silentBy.get(r.author.login) ?? 0) + 1);
  }
  const first = Math.min(...human.map((r) => Date.parse(r.submittedAt)));
  hours.push((Date.parse(pr.mergedAt) - first) / 36e5);
}
hours.sort((a, b) => a - b);
const median = hours.length ? hours[Math.floor(hours.length / 2)].toFixed(1) : "n/a";
console.log({
  mergedPrs: prs.length,
  withAnyHumanReview: reviewed,
  withWrittenFeedbackPct: reviewed ? Math.round((100 * withFeedback) / reviewed) : null,
  silentApprovalsByReviewer: Object.fromEntries(silentBy),
  medianHoursFirstReviewToMerge: median,
});

Run it monthly and keep the output. The useful signal is the trend in written feedback and silent approvals per person, not the absolute value, and a team of five will be noisy. If the number of reviewers is flat while PR volume climbs, you are reproducing the paper's setup, and the question is whether you notice before an incident does.

What to do on Monday

None of this needs an agent platform. Five steps, in order, each cheap and each reversible.

1. Run the script above on your three busiest repositories and write down written-feedback share, silent approvals per reviewer and median first-review-to-merge hours. That is your baseline.
2. Count reviewers against PR volume, not against headcount. The paper's 1.5x versus 3.1x is the ratio to watch.
3. Write down your escalation triggers: which paths, dependencies and change types always need a named human. Encode them in CODEOWNERS so they are enforced rather than remembered.
4. Turn on required code owner review and dismissal of stale approvals, so an approval cannot outlive the diff it was given on.
5. If you add an AI reviewer, run it in shadow mode first. Compare its verdicts with your humans on real PRs for a few weeks, and log every miss and every needless escalation before it is allowed to approve anything.

# .github/CODEOWNERS: escalation triggers as files, not memory.
# The last matching pattern wins, so keep the broad default first.
*                           @acme/reviewers

# Consequential changes: new service boundaries, dependencies, threat model.
/services/*/api/            @acme/architecture
package.json                @acme/architecture @acme/security
package-lock.json           @acme/security
/infra/                     @acme/platform
/.github/workflows/         @acme/platform @acme/security
/docs/threat-model/         @acme/security

# Branch protection with code owner review and stale-approval dismissal.
# All four top-level fields are required by the API; null switches one off.
cat > protection.json <<'JSON'
{
  "required_status_checks": { "strict": true, "contexts": ["ci"] },
  "enforce_admins": true,
  "required_pull_request_reviews": {
    "require_code_owner_reviews": true,
    "dismiss_stale_reviews": true,
    "require_last_push_approval": true,
    "required_approving_review_count": 1
  },
  "restrictions": null
}
JSON
gh api -X PUT repos/acme/monorepo/branches/main/protection --input protection.json

The CODEOWNERS file can live in .github, the repository root or docs, the last matching pattern wins, and one approval from any listed owner satisfies the requirement, per GitHub's documentation. The branch-protection call below uses the field names from GitHub's REST reference; all four top-level fields are required, and null disables one.

The rule I take from the paper: when output per developer doubles and reviewers do not, review does not stop, it thins, and the metrics built for human-paced pipelines will not show it. Measure written feedback and silent approvals next to merge count, write your escalation triggers down before an agent has to guess them, and treat exception-only review as a hypothesis to run in shadow mode until your own numbers earn it.