METR AI Developer Productivity Study 2026: What It Really Says

I downloaded the data.csv and regression.py that METR published for its late-2025 follow-up and ran the script unmodified on 17 October 2026. It printed minus 18.1 percent for the original developer panel, with a 95 percent interval running from minus 38.4 percent to plus 8.8 percent. That one line is the whole story of the study in miniature: a number that sounds like a verdict, sitting inside an interval that fits slower, unchanged and much faster at once.

This post reads METR's design-change post of 24 February 2026, the earlier 2025 paper, the public data repository and METR's May 2026 survey post, and attributes every figure to them. These are METR's own measurements, not independent replications, and a ten-developer subset is not your team. I draw a line between what METR wrote, what I reproduced from its published files, and what is my reading. It is not a claim that AI does or does not make developers faster.

Three numbers and one sign convention

The post, by Joel Becker, Nate Rush, Tom Cunningham, David Rein and Khalid Mahamud, is titled We are Changing our Developer Productivity Experiment Design. It reports three estimates, and only the first is a conventional finding in the sense that its interval excludes zero.

Across both late-2025 groups the post reports 57 developers, 143 repositories and more than 800 tasks, with a median of 10 years of experience. The post never states its sign convention, but the usage is consistent: the figures describe the change in task time, so minus 18 means 18 percent less time with AI. The published script computes exp of the AI coefficient minus 1 on log time, which is the same thing. Note the asymmetry: 18 percent less time is about 22 percent more work per hour, because 1 divided by 0.82 is roughly 1.22, so a time number and a throughput number are not interchangeable.

Why neither late-2025 interval is a finding

An interval that contains zero is compatible with AI making the work slower, unchanged or faster. Minus 38 to plus 9 spans 47 percentage points of task time for the original developers, and minus 15 to plus 9 spans 24 for the new ones. METR says as much: the two late-2025 intervals include zero, and only the early-2025 interval, plus 2 to plus 39 percent, excludes it.

To see how fragile that is, I ran METR's script on its CSV in a clean Python environment with pandas, statsmodels and scipy. The file holds 1,134 issues from 57 developers; the script drops 57 ditched issues and then runs the same regression twice, once on completed issues and once on issues with an initial implementation time. The first column reproduces the post.

The panel row is the lesson. Under the second filter the interval for those developers is minus 40.8 to minus 2.4 and no longer crosses zero. I am not claiming the second filter is the better one. I am saying that a headline built on about ten developers can move from compatible with zero to not, on a row-inclusion choice that never appears in the headline. Both runs are my reproduction of METR's published script, rounded as the script prints them.

The selection effect is the real subject of the post

METR names two selection problems. The first is recruitment and retention: many developers now say they would not want to do half of their work without AI, so the study loses the people most enthusiastic about it. The second is task selection: when surveyed, 30 to 50 percent of developers told METR they were choosing not to submit some tasks because they did not want to do them without AI.

METR concludes the reported estimate is likely a lower bound and that the true speedup could be much higher among the developers and tasks selected out. In the same post it says selection seems to affect a minority share of developers and tasks, and that the central estimate is likely a bad proxy for the real productivity impact of AI tools on these developers. Read together, the direction of the bias is stated and its size is not.

What survives is narrow. The late-2025 numbers describe tasks that developers agreed to do both ways, and the tasks where AI helps most are the ones most likely to be withheld. METR says these effects will only get more dramatic as capabilities and expectations grow, which is the reason it is redesigning the experiment rather than publishing a bigger sample of the same design.

Four smaller problems METR rated less severe

Beyond selection, the post lists four issues it considers less severe. Each one is a measurement trap you can fall into in a team experiment, and the right-hand column is my reading of what it means there, not METR's.

The completion gap is visible in the published CSV. Of 611 AI-allowed issues, 460 are complete (75.3 percent) and 120 incomplete; of 523 AI-disallowed issues, 368 are complete (70.4 percent) and 124 incomplete. Ditched issues are almost even, 28 against 29. The script analyses complete issues only, so the arms contribute different shares of their assignments. I am not claiming this biases the estimate in either direction; I am saying it is a number you should print.

Why asking developers does not settle it

The 2025 study already showed the problem. Before starting, developers forecast that AI would cut task time by 24 percent. Afterwards they believed it had cut time by about 20 percent. The measured result was that tasks took 19 percent longer. The gap between belief and measurement was close to 40 percentage points.

METR's May 2026 survey post looks at the question from the other side. It analysed 349 respondents, of whom 87 were software engineers, and found median self-reported changes in the value of work between 1.4x and 2x across three measures, and a median self-reported speed change of 3x that the authors expect to overstate value. The authors write that survey results are not necessarily grounded in reality and that their results plausibly suffer from significant selection bias, citing a convenience sample and a response rate of roughly 2 percent for emailed respondents.

So there are two evidence types here that disagree for understandable reasons. A survey measures perceived value across many kinds of workers; the experiment measures minutes on tasks people agreed to do both ways. Neither is an independent audit, since both come from METR. What makes the experiment checkable is that the CSV and the script are public, which let me verify the arithmetic though not the recruitment.

Measuring your own team without the same traps

The parts of METR's design that survive its own critique are cheap to copy: write the task down and the no-AI time estimate before the coin flip, randomise per task, and analyse log time with the estimate as a covariate. First reproduce METR's numbers once so you know what an interval of this width looks like.

# Reproduce METR's late-2025 numbers from its published files.
# regression.py needs: pip install pandas numpy statsmodels scipy
git clone https://github.com/METR/Measuring-Late-2025-AI-on-OSS-Devs
cd Measuring-Late-2025-AI-on-OSS-Devs
python regression.py --input-data data.csv

# It drops ditched issues, then runs the regression on all, panel and new
# developers, twice: "Total Time Merged" (status == complete) and
# "Initial Impl Started" (any issue with an initial implementation time).
# The model is OLS with HC3 errors:
#   log(totalImplementationTime) ~ ai_treatment + log(estimatedTimeWithoutAI)
# speedup = exp(beta) - 1, so a NEGATIVE number means LESS time with AI.
#
# Expect (rounded, as the script prints them):
#   Panel Developers (Total Time Merged)  -0.181  (-0.384, 0.088)
#   New Developers   (Total Time Merged)  -0.036  (-0.147, 0.09)
#   Panel Developers (Initial Impl Started) -0.24 (-0.408, -0.024)

Then run a small version on your own tickets. The script below fixes the assignment by hashing the ticket id with a seed you commit beforehand, so nobody can nudge a draw they dislike, and it reuses METR's specification with one change of mine: errors clustered by developer, since a handful of people each doing many tasks is not many independent observations.

# team_experiment.py: fixed-rule randomisation and analysis for a team trial.
# Specification adapted from METR's regression.py:
#   log(time) ~ ai_treatment + log(estimate without AI)
import hashlib

import numpy as np
import pandas as pd
import statsmodels.formula.api as smf
from scipy.stats import norm

SEED = "2026-q4-team-trial"  # commit this to the repo BEFORE the first task


def assign(task_id: str) -> str:
    # Decided from the ticket id alone, before the developer sees the task.
    # A hash cannot be nudged by someone who dislikes the draw.
    digest = hashlib.sha256(f"{SEED}:{task_id}".encode()).digest()
    return "AI-allowed" if digest[0] % 2 == 0 else "AI-disallowed"


# tasks.csv: task_id, developer_id, status, minutes, estimate_no_ai
# estimate_no_ai is logged BEFORE assignment; status is complete / incomplete / ditched
df = pd.read_csv("tasks.csv")
df["arm"] = df["task_id"].map(assign)

# Print this next to every effect you report: differential completion.
print(df.groupby("arm")["status"].value_counts(normalize=True).round(3))

done = df[
    (df["status"] == "complete") & (df["minutes"] > 0) & (df["estimate_no_ai"] > 0)
].copy()
done["ai"] = (done["arm"] == "AI-allowed").astype(int)

model = smf.ols("np.log(minutes) ~ ai + np.log(estimate_no_ai)", data=done)
# METR uses cov_type="HC3". Clustering by developer is my change: a few people
# doing many tasks each are not many independent observations.
res = model.fit(cov_type="cluster", cov_kwds={"groups": done["developer_id"]})

beta, se = res.params["ai"], res.bse["ai"]
z = norm.ppf(0.975)
change = np.exp(beta) - 1  # negative = less time with AI, same as METR's convention
low, high = np.exp(beta - z * se) - 1, np.exp(beta + z * se) - 1
print(f"Change in task time: {change:+.1%}  (95% CI {low:+.1%} to {high:+.1%})")

METR's script uses HC3 robust errors and treats every issue as independent. Clustering by developer is my addition, not METR's, and with fewer than about ten developers it will not rescue the sample; it will only make the interval honestly wide. Treat the output as a prompt for a conversation, not a scoreboard.

What METR is changing, and what each fix costs

METR says wider adoption of agentic tools such as Claude Code and Codex made recruitment harder and changed which tasks developers submit. Its planned responses are below. The cost column quotes METR where it states one and is marked as my inference where it does not.

METR also says it expects to keep redesigning parts of its measurement as model capabilities increase. That is the useful signal for an engineering manager: the leading source on this question has told you its own instrument is wearing out, so a single number from it, or from you, deserves a standing expiry date.

What to do on Monday

You do not need a research programme. You need a rule you commit to before the data arrives, and the habit of printing the interval and the completion rate next to every effect you report.

1. Run METR's regression on its published CSV once, so the width of a 10-developer and a 44-developer interval is something you have seen rather than read.
2. Pick one class of work, such as tickets that already carry an estimate, and write the inclusion rule down, including what happens to ditched and abandoned tasks.
3. Log the no-AI time estimate before assignment, and assign by hashing the ticket id with a seed committed to the repository.
4. Record time to first implementation and time spent on review and rework separately, the way the METR dataset splits them, and note when the person was multitasking.
5. Report the interval and the completion rate per arm, never a point estimate alone, and put a re-measure date on the result.

When someone forwards the headline as AI makes developers 18 percent faster, the accurate reply is short: the estimate is 18 percent less time on a ten-developer subset, the interval runs from 38 percent less to 9 percent more, and the authors call the central estimate a poor proxy.

The rule I carry from this: an AI productivity number is only as good as its interval and its selection story. METR's update is valuable because it prints both, and an honest reading is that the late-2025 data neither confirm nor rule out a speedup, while the early-2025 slowdown was measured on a different generation of tools. Measure your own work with a rule fixed in advance.