Agents Did the Engineering of Research and Scored 2 of 6

Agents Ran Hundreds of Experiments, Wrote the Paper, and Got 2 Out of 6

Princeton and the UK AI Security Institute gave an agent the central question of an unpublished NeurIPS submission, then had the paper's real authors grade the result as reviewers. Both attempts were unambiguous rejections.

Jul 30, 2026Source: arXiv

A team spanning Princeton, the UK AI Security Institute, Cornflower Labs, Georgetown and others published Can AI agents conduct open-ended AI research? on 29 July. The method is the interesting part. They took two unpublished NeurIPS 2026 submissions, gave an agent each paper's central research question, and had the papers' actual authors grade the results as conference reviewers.

The setup

Claude Opus 4.8 with extra-high reasoning on the OpenClaw scaffold, six days of wall-clock time, $3,000 in API credits, GPU credits, a Linux VM and the open web.

The engineering worked

The agents completed large literature reviews, debugged GPU environments, ran hundreds of experiments and robustness checks, and compiled full camera-ready LaTeX documents with no manual intervention beyond credentials and repository setup. They resolved every environmental obstacle but one. Surveyed in advance, nine of eleven coauthors expected the runs to die in an unresolvable error loop — they were too pessimistic.

The research did not

Author reviews on NeurIPS scales: the first paper scored 2/6 overall, the second 1/6, with reviewer confidence of 4 and 5 out of 5. Both were unambiguous rejections. The named failure modes: poor judgment about the bar for publishable research, uncreative responses to design shortcomings, ineffective backtracking from dead ends, poor resource awareness, and instruction drift.

The most telling detail

Both runs ended with under half the API budget spent despite explicit encouragement to use it, and both retired their most ambitious targets within the first ten hours and never revised course. One draft carried zero main-body visualisations against 15 in the original. A rerun on GPT-5.6 Sol reproduced the failures — and burned the entire $3,000 in two days.

Why It Matters

The automated-research debate has been argued from benchmarks that reward well-specified tasks. This measures the other thing: taste, ambition and knowing when work is good enough. Agents cleared every engineering obstacle and still could not tell that what they had produced was not publishable.