Better Refusal Tests Begin with Nearly Identical Questions

A September 8 article from Multiverse Computing argues that refusal behaviour should be evaluated at the boundary between permitted and prohibited requests, not merely across broad topics. Its study uses political prompts to examine how a model can learn to reject manipulative requests while still answering legitimate factual questions. The authors report that improving refusal on harmful examples can also produce substantial over-refusal on safe ones.

That observation has consequences well beyond a particular training recipe. A support assistant that declines ordinary account questions is not doing its job, even if its harmful-response rate is low. Conversely, an assistant that answers every request is not reliable simply because users rarely encounter a refusal. Both outcomes need to be visible in an evaluation.

Define what successful assistance looks like

Start with the work the product should perform. For an illustrative public-information assistant, useful tasks might include locating an official opening time, explaining an application form or summarising a published eligibility rule. Describe the evidence an answer needs and where the assistant should ask for clarification rather than guessing.

Next, describe the prohibited behaviour at the level of actions and intent. Broad words such as sensitive or controversial are too ambiguous to make dependable test labels. Reviewers should be able to explain why a request falls on one side of a boundary without relying solely on a keyword in its wording.

This process often reveals cases that are neither straightforward compliance nor refusal. A user may have a legitimate goal but provide insufficient context. An assistant can ask a focused question, give a narrower explanation or refer to an authoritative source. Evaluation should recognise these useful responses rather than forcing every answer into an oversimplified binary category.

Build pairs that isolate the real distinction

A paired test changes one meaningful feature while keeping the surrounding topic similar. For example, compare a request to explain an organisation's published access policy with a request to bypass that policy. The shared vocabulary makes it harder for a superficial keyword rule to appear competent. The expected difference is grounded in the requested action.

Write each pair before looking at the model's output. Otherwise reviewers may unconsciously adapt their interpretation to whichever behaviour the model already exhibits. Include a short rationale with each expected answer so that another reviewer can disagree constructively and improve the policy definition.

Avoid measuring only obvious examples. Real users make spelling mistakes, omit context, mix languages and ask follow-up questions. Include these variations when they reflect the intended audience. Keep some pairs genuinely unseen until the final evaluation; repeatedly tuning against the entire test set turns a useful measurement into a memorisation target.

Report two kinds of failure separately

Count harmful compliance and unnecessary refusal independently. A single combined score can conceal a tradeoff: one version may look better overall because it refuses more of everything. Show the results for both members of each pair, then group them by the kind of distinction being tested.

Also inspect the content of the answer. A response can begin with a refusal and subsequently provide what it claims to withhold. A supposedly helpful answer can avoid the question entirely. An automatic judge may help triage large collections, but a smaller human-reviewed sample is useful for detecting these mismatches between labels and actual behaviour.

When reviewers disagree, preserve the disagreement. It may indicate an unclear policy rather than a model failure. Record the reason, resolve the product decision where possible, and keep a separate set of unresolved cases. Quietly dropping difficult examples would make the evaluation easier while making the deployed system no more dependable.

Treat updates as changes to a decision boundary

After changing a prompt, guard or fine-tuning dataset, rerun the permitted and prohibited examples together. Compare which individual cases changed, not only aggregate percentages. A model that fixes one failure by introducing another nearby needs a different release discussion from one that improves both sides.

Keep representative ordinary tasks in the same review. Safety adjustments can affect tone, clarification behaviour and willingness to complete routine work. These effects matter to users even when no policy boundary is being approached. A rollback plan should preserve the previous version and its evaluation record rather than depending on memory of how it seemed to behave.

The practical goal is selective reliability. An assistant should help confidently when the request is appropriate, identify uncertainty when context is missing, and decline the specific behaviour that crosses a defined boundary. Measuring those distinctions is more informative than celebrating a larger pile of refusals.