METR’s famous coding slowdown result now comes with a harder measurement problem

METR’s July 2025 study randomized 246 real issues across 16 experienced open-source developers working in repositories they knew well. With early-2025 AI tools available, they took 19% longer. They had expected a 24% speedup and still believed afterward that AI had made them faster.

That result was specific to those tools, tasks and developers. It did not establish that AI slows all software work, and METR’s later experiment complicates attempts to carry the number forward.

In February 2026, METR reported that developers increasingly declined to participate because they did not want to work without AI. The researchers also reduced compensation from $150 to $50 an hour. Both changes could alter who joined and which tasks entered the study. Concurrent agent use made time measurement harder too.

Raw estimates suggested tasks took 18% less time for returning developers and 4% less for newly recruited developers. Both confidence intervals included no effect, and METR judged selection problems too severe to treat the figures as reliable estimates of the current benefit.

The researchers think acceleration has probably increased, but cannot confidently quantify it from this design. The combined record separates an experimentally observed historical slowdown from a changing present in which measuring the counterfactual has itself become difficult.