Untrained Model Scores 0.280 on Self-Improvement Metric

An Untrained Model Scored 0.280 on the Statistic Used to Prove Self-Improvement

Auditing three rounds of LoRA self-training against a frozen control, four researchers found seven measurement failures — each of which inverts a published finding when the control is missing.

Aug 21, 2026Source: arXiv

A paper submitted to arXiv on 20 August audits self-improvement claims by doing the thing such claims usually omit: running a control. The authors put three rounds of rank-32 LoRA self-training on Qwen3-8B against a frozen model pushed through the identical pipeline, and identify seven measurement failures, each of which inverts a reported finding when its control is absent. Several, the authors note, are standard practice.

The central number

Self-improvement is increasingly judged not by mean accuracy but by which individual problems a model gains and loses — a framing that requires differencing two noisy estimates. A ledger built on a single greedy decode manufactures capability changes on a model that was never trained, largely as an artefact of inference batching. And the expansion statistic used to separate genuine acquisition from mere sharpening assigns that same untrained model a rate of 0.280. The metric reports substantial self-improvement in a system where, by construction, none occurred.

What the common framing gets wrong

Results in this literature are reported as before-and-after comparisons on the same model, which reads as its own control. It is not. Between the two measurements sit sampling noise, decoding non-determinism and batching effects, all of which produce per-problem transitions with no training involved. The obvious repair — set a threshold high enough to exclude noise — does not survive replication: estimated across the frozen comparisons a design already contains, the null stays non-zero. The authors replace it with a per-problem exact test against a pooled baseline under false-discovery-rate control.

Scope, stated honestly

This is one model family at one scale with one adaptation method. It does not show that self-improvement is illusory. It shows that the instruments commonly used to detect it register a strong signal on a system that cannot have produced one — which means published effect sizes cannot be trusted to be larger than the measurement floor, because the floor was never measured.

Why It Matters

Recursive self-improvement is among the most consequential claims in the field and among the hardest to measure. A control run showing the standard metric fires on an untrained model puts the burden back on every result that skipped one.