Zero-Token Confidence Turns Hidden States Into a Lightweight Model Gate

A language model can produce a fluent answer and still be wrong. The usual response is to ask the model to critique itself, call a second judge model, or sample several answers and compare them. Each option consumes generation time and tokens. FINAL-Bench's new Zero-Token Confidence (ZTC) release explores a different interface: read an internal activation from a completed forward pass and map it to a calibrated estimate of correctness.

The idea is operationally appealing because it separates answering from verification. Instead of generating an explanation of confidence, a small learned probe examines the final hidden state. The published probe package contains a weight vector, normalization statistics, and calibration constants, rather than another generative model. That makes ZTC less like a conversational critic and more like a compact sensor attached to a particular model.

What the release reports

FINAL-Bench reports results over a shared 2,018-item benchmark spanning multiple domains. Its top listed ZTC system, built around Darwin-397B, reaches an AUC of 0.7394 without generated verification tokens. A ZTC judge based on a 27B model reaches 0.7255. The article also includes a deliberately simple surface baseline using answer length and formatting, which scores 0.7036. That baseline matters: any verifier below it may be exploiting less useful signal than cheap metadata already provides.

For self-readout on 539 held-out items, the release reports AUC improvements from 0.7646 to 0.8801 for Darwin-397B and from 0.6141 to 0.8388 for Qwen3.5-27B when switching from prompted confidence to hidden-state readout. It also reports a leave-one-domain-out score of 0.7572 for Darwin-397B. These are the authors' benchmark results, not a guarantee that the same probe will transfer to a new task, model revision, prompt format, or production traffic mix.

The latency numbers are similarly specific to the disclosed setup. On four B200 GPUs with a 27B model and 2,000 items, the article reports 0.0615 seconds for the gate versus 1.631 seconds to generate one candidate answer. The comparison supports the case for testing a hidden-state probe where generation is the dominant cost, but it should not be treated as a universal speedup. Hardware, batching, sequence length, activation access, and serving architecture can all change the balance.

A probability is useful only with a decision rule

AUC measures ranking: whether incorrect and correct outputs tend to receive different scores. A product, however, needs a threshold and an action. Below the threshold, should the system abstain, ask for human review, retrieve more context, or call a stronger verifier? Each choice has a different cost.

Consider an agent that proposes database updates. A team could reserve automatic execution for scores above a high threshold, send a middle band to a conventional verifier, and block the lowest band. The important calculation is not simply whether ZTC has the best aggregate AUC. It is whether the selected threshold catches enough costly errors without rejecting so many correct actions that the workflow becomes unusable.

Calibration must also be checked directly. If outputs assigned 0.8 confidence are correct only 60% of the time, the score may still rank examples well while misleading downstream policy. Reliability diagrams, expected calibration error, and per-domain precision and recall reveal properties that a single leaderboard number cannot. Thresholds should be chosen on validation data that reflects the deployment's error costs, then frozen before final evaluation.

Test the probe as a coupled component

A hidden-state probe is coupled to the representation it reads. Model weight updates, quantization, changes to chat templates, altered tokenization, or moving the readout to another layer can shift that representation. Teams should version the base model, probe, prompt template, extraction point, and calibration data as one artifact. A compatibility check belongs in the rollout process.

A robust evaluation should include at least three comparisons: the ZTC probe, a no-gate baseline, and a cheap surface baseline. Add a generative judge if that is the current production approach. Run every method on identical examples, report confidence intervals, and separate domains rather than relying only on an average. Stress tests should include ambiguous questions, malformed inputs, distribution shifts, and confidently stated false answers.

It is also worth measuring total system cost. Reading activations may require changes to an inference stack that normally returns only logits or text. Retaining hidden states can affect memory pressure and batching. A tiny probe file does not automatically imply a tiny integration burden, particularly with hosted APIs that do not expose internal states. ZTC is therefore most directly applicable when the operator controls model inference or uses an endpoint designed to expose the required activation.

Where zero-token gating fits

ZTC does not replace factual verification. It does not retrieve evidence, identify the specific flaw in an answer, or prove that an action is safe. Its strongest role is triage: cheaply decide which outputs deserve more expensive scrutiny. A fast gate can make cascade designs practical, allowing the majority of clear cases to pass while routing uncertain cases to retrieval, a judge model, or a person.

The release is a useful reminder that generated self-assessment is only one source of confidence information. Internal representations may carry a stronger correctness signal, but extracting that signal responsibly requires model-specific fitting, careful calibration, and repeated validation after system changes. The promising question is not whether a 45 KB probe can replace every verifier. It is whether such a probe can make a layered verification policy faster while preserving the error tolerance the application actually needs.