Off-by-1 Labs, 1Password's research group, published results on 6 August from grading 6,080 AI-generated security patches. Only 26.0% fully fixed the vulnerability without changing application behaviour. The authors had hypothesised a rate above 67%.
The other 74% is not all failure
A further 20.1% closed the vulnerability but altered how the application behaved — so 46.1% of patches did fix the bug. The remaining 53.9% failed to remediate, introduced a new vulnerability, or both. Only 2.3% introduced a new security issue while fixing the original. The headline number understates the fix rate and overstates the danger.
Method
Six recently disclosed CVEs were chosen specifically to avoid training-data contamination — a model cannot have memorised a fix published after its cutoff. 6,480 patches were generated and 400 flagged and removed. The models were OpenAI's ChatGPT-5.5 and Anthropic's Opus 4.8, at an average $2.11 and $2.81 per patch respectively. The authors are Axel Mierczuk, Spencer Michaels and Keith Hoodlet.
Fragile even when right
More than a third of the patches that succeeded were rated "fragile" by the authors — correct, but resting on assumptions that would not survive a refactor. The widely quoted $6.74 figure is cost per successful patch, a different metric from 1Password's per-patch generation cost, and the gap between the two is the tax paid on the discarded attempts.
What it does and does not measure
This is a six-CVE benchmark on unreviewed output, not production telemetry, and the models tested are not the current frontier releases. What it measures precisely is the case the industry keeps proposing: patches applied without a human reading them. On that specific question the gap between the authors' hypothesis and the result — 67% expected against 26% observed — is larger than the result itself.