New Coding Benchmark: 5.4% Passed, Not The Quoted 47/100

A paper posted on Monday argues that existing agent benchmarks score only behavioural correctness, which lets an agent pass a migration task by quietly copying the original implementation rather than migrating it. The authors name this failure mode "Blindness" and build a three-stage protocol to catch it: a migration audit, then behavioural tests, then agentic verification by six independent coding agents generating targeted tests.

The headline result

Across 520 runs, drawn from 8 frontier models in 26 model-effort configurations, on 20 whole-repository migrations covering four kinds of technical debt: 28 runs — 5.4% — pass all three stages. 13 of the 20 tasks receive no accepted solution from any configuration. By category, build-toolchain rewrites score 31.4 and language rewrites 5.6.

What the conventional framing gets wrong

Two numbers from the same runs are about to be used interchangeably. 47.0 out of 100 is the best model's composite score on the paper's own scale. 5.4% is the all-three-stages pass rate. Quoting 47/100 as a success rate overstates by roughly nine times; quoting 5.4% as that model's score understates it just as badly. The second trap is the paper's most quotable line: among the 340 runs that clear the migration audit, 58% reach 99% of the fixed checks but only 26% reach 100%. That reads like near-success, and the paper's entire point is that near-success is not success — a migration that is 99% complete is a migration that does not build.

The caveat that belongs in the first paragraph

This is a self-built benchmark with self-reported numbers, published on day one, with no external replication. The authors define the tasks, define the scoring, and set the three-stage gate — and the "agentic verification" stage is itself performed by coding agents. The ranking should be read as the authors' construction, not as a settled measurement of anything.

Why it is still worth reading

The claim that agents will pay down technical debt rests entirely on long-horizon, repo-wide work, and that is the hardest thing to measure precisely because behavioural equivalence is cheap to fake. A gate designed specifically to catch the copy-the-original shortcut is a useful instrument even if its first set of numbers turns out not to hold.