Layer Feedback Separates Transformer Size From Execution Depth

A different way to spend a parameter budget

Layer-Feedback Transformer (LFT) proposes changing the path through an otherwise conventional Transformer stack. Instead of executing every stored block once, it revisits adjacent blocks before moving deeper. A five-block example therefore follows an eleven-execution route, while retaining only five unique blocks. The source describes standard ingredients including causal attention, RoPE, RMSNorm, SwiGLU, residual connections and tied embeddings; the routing graph is the defining change.

For N unique layers, the proposed route performs 3N - 4 block executions. That distinction is the useful idea: parameter depth describes how many different blocks are stored, while execution depth describes how many transformations a hidden state undergoes. Conventional stacks usually make those quantities equal. LFT deliberately separates them.

Fewer parameters does not mean less work

Parameter count is only one resource constraint. Reusing weights can reduce the number of distinct values that must be stored, transferred or optimized, but each reuse still invokes attention and feed-forward computation. The execution multiplier approaches three as the number of unique layers grows. Activations, latency, energy use and training time therefore remain central deployment concerns.

This makes LFT most relevant where model storage is a harder limit than computation. A device might have room for a compact set of weights yet enough time to execute those weights repeatedly. The opposite environment—a latency-sensitive service with ample memory—may prefer a conventional model that performs fewer sequential block calls. Real suitability depends on the entire runtime profile, not the parameter number printed beside a checkpoint.

Read the experiment along two axes

The reported study compares standard and feedback-routed models at roughly 2.5 million, 10 million and 25 million parameters. Each pair shares its unique-layer count, training stream and 500-million-token budget. LFT trails at the smallest scale, leads more clearly on the source's aggregate Base Bench result at 10 million parameters, and has a smaller lead at 25 million. An earlier 10-million-parameter run trained for 200 million tokens had favored the standard model instead.

Those observations support a limited conclusion: the routing deserves further study, and its outcome can depend on scale and training duration. They do not establish that feedback routing is generally superior. Three small configurations, a few evaluation sets and one training-data regime leave many unanswered questions about larger models, other corpora and result stability.

The missing comparison is compute matched

The pairs are parameter matched and token matched, but they are not matched for floating-point operations. For the tested layer counts, LFT executes blocks more than twice as often for every token. A score improvement could arise from the route itself, from the added computation, or from an interaction between both.

A stronger evaluation would add at least two baselines. First, compare against a standard Transformer allowed a similar total training-compute budget, even if that means changing its parameter count or number of tokens. Second, compare against simpler forms of weight sharing or repeated application with the same number of block executions. These controls would help isolate whether the neighboring feedback pattern adds value beyond merely doing more sequential work.

Wall-clock measurements matter too. A theoretical block count does not capture kernel efficiency, memory traffic or compiler behavior. Report training throughput, peak memory, inference latency and energy under the same hardware and software settings. Because repeated blocks form a longer sequential dependency chain, they may be difficult to parallelize away.

Test whether reuse changes optimization

A shared block receives inputs produced at different points in the route. Its weights must serve several computational roles rather than specialize for one fixed depth. That reuse could regularize learning, create interference, or do both at different stages. The reversal between the shorter and longer 10-million-parameter runs makes training curves particularly important.

Future experiments should preserve validation checkpoints across training, repeat runs with multiple seeds and report variance. Gradient statistics for repeatedly used blocks could show whether particular passes dominate learning. Ablations might vary the number of revisits, share only selected blocks, or stop feedback at certain depths. Such tests would reveal whether the proposed schedule is robust or tuned to one routing pattern.

A practical evaluation checklist

Start with the constraint the architecture is meant to solve. If weight memory is the bottleneck, record model-file size and resident memory. If latency or energy is the bottleneck, measure those directly instead of using parameter count as a proxy. Keep tokenizer, data order, context length and evaluation prompts fixed when comparing routes.

Then inspect behavior by task rather than relying only on an average. Improvements in contextual or quantitative questions may coexist with regressions elsewhere. Include confidence intervals or repeated runs where feasible, and keep the exact checkpoint and evaluation configuration traceable. Finally, test long contexts separately: repeated attention blocks can make the compute tradeoff increasingly significant as sequences grow.

LFT is best understood as a proposal for allocating capacity: store fewer distinct transformations, apply them more often, and accept extra sequential compute. Its early results make that tradeoff worth testing, but a deployment decision should wait for compute-matched baselines and measurements on the hardware, context lengths and tasks that actually matter.