A benchmark paper posted on 3 September takes aim at how the field scores its flagship security claim. If an AI agent patches a vulnerability, the standard check is to rerun the crashing proof-of-concept and see whether it still crashes. The authors find that gate is loose by a wide margin.
The headline measurement
"Across 11 state-of-the-art agents, including the top three AIxCC agents, the original PoC-only validation inflates the patching task solve rate of agents by 1.83x on average." The work covers C and C++ vulnerability patching, and the comparison is between two validation regimes rather than between two sets of agents.
How the benchmark is built
It "selects vulnerabilities whose ground-truth fixes lie outside the crash stack", then applies vulnerability transplant and code mutations. That construction matters for reading the number: these are cases chosen precisely because a patch can silence the crash without addressing the defect.
The memorisation finding
Separately, "on average, 25% of the agent patches exhibit substantial similarity to historical developer patches." That is a similarity measure, not proof of training-data contamination — a correct fix often resembles the developer's fix because there is one obvious correct fix — but it is a quarter of the sample, and it bears on what these benchmarks are measuring.
What the received framing gets wrong
The headline this will attract is that AI patching agents are 45% fake. That is not what 1.83x says. The ratio compares two scoring methods on a benchmark the authors built specifically to break the weaker one, so the gap is engineered to be visible rather than sampled from the wild. The defensible reading is directional: PoC-only validation systematically overcounts, by roughly this much on adversarially chosen cases. The abstract also releases no per-agent scores, so no vendor can be singled out from it. The limits are worth stating — one team, one language pair, a benchmark of the authors' own construction, and no independent replication yet.