Agent Reproductions Need Better Questions, Not Just More Runs

Research reproducibility has usually been constrained by attention. A reviewer can inspect a proof, rerun an experiment, or audit a dataset, but rarely all three across every submission. Coding agents change that constraint: they can turn papers into executable checks, run many investigations in parallel, and preserve the resulting artifacts. The important question is no longer whether agents can attempt reproductions. It is how to design those attempts so their outputs deserve confidence.

A Hugging Face community challenge around ICML 2026 offers a useful case study. Over 19 days, more than 1,200 participants published 6,816 logbooks covering 2,226 papers. An automated judge assigned claim-level verdicts, while the logbooks exposed code, artifacts, and sometimes complete agent traces. The scale matters, but the deeper lesson is methodological: a reproduction pipeline is only as strong as its targets, stopping rules, and review layers.

Start With Claims, Not Papers

A paper is too broad to be a single test unit. Its central result may depend on a theorem, a preprocessing choice, a baseline configuration, and several empirical comparisons. Declaring the whole paper “reproduced” can hide which link was actually checked.

A better workflow begins by converting the paper into bounded claims. For each claim, record the expected observation, required inputs, acceptable tolerance, and conditions that would make the result inconclusive. This creates a test specification before the agent starts coding. It also makes partial success informative: confirming one experiment does not silently validate an unrelated proof or data claim.

Claim extraction introduces its own risk. An agent may simplify away a qualifier such as “under convexity” or “at this model scale.” A human review of the claim list is therefore high leverage. It is cheaper to correct a distorted target before compute is spent than to untangle a polished experiment that answered the wrong question.

Treat Every Reproduction as an Evidence Package

A verdict without provenance is not very reusable. A strong evidence package should connect four layers: the paper’s claim, the reproduction code, the produced artifacts, and the reasoning that maps observations to a verdict. Environment details and random seeds help another investigator distinguish a scientific disagreement from a software mismatch.

The challenge’s public logbook pattern points toward a practical standard. Keep raw outputs alongside summaries; identify unavailable datasets or checkpoints; label reduced-scale tests as toy evidence; and preserve failed approaches when they reveal ambiguity. “Inconclusive” is a valid result when prerequisites are missing. Pretending that synthetic data establishes a full real-world claim is not.

Automated judges can organize this material, but they should not be the final authority. The source report describes both contested claims and a reproduction whose dramatic performance conclusion came from comparing incompatible timing units. A judge should independently read the evidence, yet its verdict still needs challenge paths and spot checks.

Design Tests That Can Prove You Wrong

Many weak reproductions ask whether a familiar plot can be recreated. Stronger ones search for the boundaries of a claim. If a theorem describes asymptotic behavior, extend the horizon rather than stopping where the curves look reassuring. If a result depends on a loss function, compare the equation, implementation, and experiment configuration. If an aggregate metric can be diluted by padding, inspect the composition of evaluated examples.

Independent attempts are especially valuable because disagreement is diagnostic. Opposite verdicts may expose sensitivity to seeds, dependencies, interpretation, or scale. The next step should not be majority voting. It should be a differential audit: compare assumptions and execution paths until the divergence is explained.

Teams can use a simple escalation rule. One successful run supports further testing. Two genuinely independent runs increase confidence. Conflicting runs trigger targeted comparison. A surprising falsification gets adversarial reimplementation before anyone contacts authors or publicizes a conclusion. This approach spends human attention where it changes the outcome most.

Keep Humans at the Decision Boundaries

Agents are well suited to setup, code generation, parameter sweeps, artifact collection, and consistency checks. Humans remain essential where the task involves scientific intent, perceptual quality, or the meaning of an unexpected result. The useful division of labor is not “agent executes, human approves.” It is a sequence of explicit checkpoints.

Before execution, a researcher reviews the claim and test design. During execution, intervention is warranted when the agent loops, changes the question, or encounters missing resources. After execution, a reviewer challenges the inference, checks units and scales, and decides whether domain judgment is needed. Image quality, clinical relevance, and social context cannot always be reduced to a convenient automatic score.

A More Scalable Review Model

Agent-assisted reproduction will not solve overloaded peer review by producing more verdicts alone. It can help by producing structured, inspectable evidence and directing scarce expert attention toward disagreements, boundary cases, and high-impact claims.

The durable outcome of a reproduction effort should be more than a green or red label. It should be an audit trail that another team can rerun, criticize, and extend. At conference scale, that turns review from a one-time judgment into an evolving evidence process—and gives both humans and agents roles matched to what they do best.