Small Local Models Become More Useful When the Harness Can Learn

A capable model surrounded by smaller local workers sounds like an obvious route to cheaper, parallel software work. A recent community experiment makes the more interesting point: the value of that arrangement depends less on the number of agents than on whether the environment turns failures into reusable guidance.

The experiment asked agents to recreate Python behavior from compiled programs. One task contained 38 functions for an animated pixel-art logo; another contained 56 functions for a small space-shooter game. A DeepSeek model acted as lead, while four Qwen 9B workers handled individual functions. Every proposed rewrite faced a differential-testing oracle that compared outputs and state bytes with the original program.

Both team runs eventually accepted every function, but the smaller task favored a solo lead by a wide margin. On the larger task, team and solo costs became much closer, although the team still took more active time. Those results do not establish a general scaling law. They do, however, expose several design lessons that apply well beyond binary analysis.

Verification Makes Delegation Concrete

Delegation works best when success can be stated mechanically. Here, a worker did not finish because its code looked plausible or passed a friendly example. It finished only after matching the original across held-out cases, including every relevant byte of state. That tight oracle changed the lead-worker relationship: the lead could assign bounded units of work without manually reviewing every line.

This pattern transfers to other domains when an equally sharp contract exists. A data transformation can be checked against invariants and fixtures. A protocol client can be tested against recorded exchanges. A UI component can be compared through accessibility trees and deterministic screenshots. The worker's model size becomes only one variable; test quality, task boundaries, and diagnostic feedback may dominate.

The converse matters too. If the evaluator measures only whether code runs, agents can converge on superficially successful implementations that miss edge behavior. Before adding workers, teams should ask whether the target can be decomposed into independently testable units and whether failures reveal enough information to guide the next attempt.

The Lead Should Improve the System, Not Just Rescue Tasks

In the first experiment, workers escalated 15 of 38 functions, and the lead changed none of their tooling. In the second, 20 of 56 functions escalated, but the lead also revised worker instructions and added a lookup tool. Repeated mistakes included signed-versus-unsigned branch interpretation, incorrect function signatures, and wrong memory offsets. Encoding those lessons let workers solve later problems without another manual rescue.

That distinction is important. A lead that merely completes failed assignments is an expensive fallback worker. A lead that identifies a recurring failure class and changes prompts, context, tools, or validation improves the throughput of every future assignment. The latter resembles maintaining an internal platform: observe failure patterns, remove ambiguity, and make the correct path easier to follow.

A useful operational rule is to track escalation reasons rather than only escalation counts. If five workers fail for the same reason, the next action should probably be a shared harness change. If each failure is unique and requires deep global reasoning, the decomposition itself may be wrong.

More Agents Do Not Guarantee Better Economics

The 38-function team run cost $2.97 and took 86 active minutes, while the solo run cost $0.87 and took 14 minutes. For the 56-function game, the team cost $3.60 and took 100 minutes, compared with $3.28 and 54 minutes for the solo approach. These are two toy workloads, so the numbers should be treated as observations, not forecasts.

Still, they offer a practical evaluation framework. Compare at least four dimensions: total compute cost, wall-clock latency, lead-model usage, and completion reliability across repeated runs. Also count the engineering cost of building the oracle and maintaining the shared task state. A worker pool can look inexpensive per token while creating coordination overhead that erases the saving.

Task granularity is another constraint. Units that are too broad overload small models; units that are too narrow multiply setup costs and dependencies. Good candidates have a compact context, a stable interface, a deterministic check, and limited need for knowledge held by neighboring tasks.

A Better Pilot for Local Agent Teams

Start with a representative slice rather than a full migration. First, define the artifact and evaluator before selecting models. Next, run a strong solo baseline using the same tools and tests. Then add a small worker pool and record every retry, escalation, context transfer, and harness modification. Finally, rerun after the harness changes: improvements that cannot be reproduced may reflect an easy sample rather than a better system.

Pay special attention to whether guidance accumulates. A lookup utility, an explicit schema, or a diagnostic that names the mismatched field can be more valuable than a longer prompt. Conversely, continually expanding instructions can burden workers with irrelevant rules. Promote only recurring, verified lessons into the shared environment.

The Main Result Is an Adaptive Scaffold

The experiment does not show that small local models can replace a strong model for arbitrary software reconstruction. It shows a narrower and more useful possibility: small models can contribute exact work when assignments are bounded, feedback is precise, and a lead can repair the environment that shapes their behavior.

For long-running projects, that adaptive scaffold may be the real compounding asset. Models and task queues are replaceable. A harness that captures interfaces, exposes trusted facts, diagnoses errors, and converts repeated failures into better tools can make the whole system progressively more capable.