SafetyPulse Treats Fine-Tuning Safety as a Checkpoint-by-Checkpoint Signal

Fine-tuning an aligned language model creates a deceptively simple operational question: how can a team tell whether useful adaptation is also weakening safety behavior? Evaluating only the final checkpoint can reveal a problem too late, while a small refusal test may be too noisy to serve as an alarm. SafetyPulse, presented in a Hugging Face community article, explores a different approach: monitor internal refusal-related behavior throughout training and reserve a broader behavioral audit for confirmation.

The work is a case study on Cohere Labs’ 3.35-billion-parameter Tiny Aya Global model. Its central claim is not that one internal score certifies safety. Rather, a circuit-derived measurement can act as an early screening signal during fine-tuning, helping teams decide when to stop, investigate, or repair a run. That distinction matters because model safety cannot be reduced to a single dashboard number.

From a final audit to a training-time alarm

SafetyPulse begins by calibrating a refusal signal on matched harmful and harmless prompt pairs. The authors identify a refusal direction and use attribution patching to rank attention heads and MLPs by their contribution to the model’s refusal behavior. Their selected circuit contains 14 MLPs and six attention heads across later layers.

For each checkpoint, the system computes a Circuit Integrity Score (CIS) using 50 probe pairs. The primary circuit score compares the attributed contribution of those components with their contribution in the base model. Lower values indicate degradation, and fixed thresholds map the result to PASS, WARN, or FAIL. Non-finite values are treated as errors rather than silently passing.

This design turns checkpoint monitoring into triage. A warning does not prove that the model has become unsafe, and a pass does not certify it. Instead, the score indicates whether the fine-tune has altered an internal mechanism enough to justify behavioral testing. In the reported held-out runs, the monitor flagged every unsafe checkpoint and warned one to three checkpoints before measurable harm appeared. However, its precision was 0.46, including a warning on a GSM8K fine-tune that did not reduce safety. That false-positive rate reinforces the intended role: alarm first, audit second.

Why orientation alone can miss fading behavior

A useful conceptual contribution is the separation between the direction and the strength of a representation. A cosine-similarity monitor measures whether an activation difference still points in roughly the same direction. If the vector shrinks without rotating, cosine similarity can remain unchanged even as the underlying separation between harmful and harmless inputs weakens.

The circuit score is designed to respond to more than orientation. In the authors’ experiments, it correlated more strongly with attack success rate than direction-drift scores calculated from the same activations: a reported Spearman correlation of −0.80 versus about −0.56. A 50-prompt refusal spot-check caught only 13% of unsafe checkpoints under the study’s alarm rule.

This does not establish that circuit monitoring will outperform behavioral checks on every model. It does show why teams should examine what a metric is mathematically capable of detecting. A stable direction, a stable norm, and stable behavior are different properties; monitoring one should not be mistaken for monitoring all three.

Detection and repair are separate problems

The study also tests where safety drift is stored. Restoring only the identified circuit’s weights did not improve attack success rate. The authors interpret this as evidence that the circuit reads out the drift but does not necessarily contain the parameter changes responsible for it. That is an important operational boundary: localization for monitoring does not automatically reveal the correct repair target.

The most effective reported repair was SafeTune’s rank-8 safety-vector restore, a low-rank approximation of the update back toward the base model. Across three drifted Alpaca runs, attack success rate fell from 25.2% to 12.3%, while GSM8K accuracy rose from 55.1% to 70.7%. Whole-model interpolation reached the same average attack success rate but recovered less GSM8K performance. Steering reduced harmful compliance too, though with a larger capability tradeoff and more over-refusal.

These comparisons illustrate why a repair must be evaluated on several axes. Lower attack success is insufficient if the model begins refusing benign requests or loses the task capability that motivated fine-tuning. SafetyPulse therefore reports HarmBench attack success, XSTest over-refusal, GSM8K accuracy, and MMLU performance rather than optimizing a single headline metric.

A practical evaluation checklist

Teams considering a similar workflow should treat calibration as model-specific. The reported circuit, layer choice, and thresholds were derived for one base model; the authors explicitly list single-model coverage as a limitation. A sensible evaluation would therefore ask:

• Are thresholds fixed before scoring held-out training runs?
• Does the alarm lead behavioral degradation rather than merely track it afterward?
• How often does it warn on capability fine-tunes that remain behaviorally safe?
• Are repairs compared against simple controls, including interpolation and unrelated component restoration?
• Do audits measure harmful compliance, benign over-refusal, target-task quality, and broader capability?
• Are uncertainty intervals and paired tests reported for behavioral changes?

The study also found that removing 19 examples through data curation did not prevent drift in its Alpaca condition. This should not be read as an argument against data hygiene. It means curation and checkpoint monitoring address different risks: clean data can still produce an update that changes safety behavior.

The right takeaway: instrumentation, not certification

SafetyPulse offers a concrete prototype for making safety review part of the training loop. Its strongest idea is procedural: calibrate once, monitor every checkpoint, escalate warnings to behavioral audits, and validate any repair across both safety and capability measures.

The limits remain substantial. The experiments cover one base model and one benign drift condition, direct harmful requests rather than jailbreak attacks, and relatively small behavioral test sets. The judge was compared with another automated judge, not human labels. Those constraints prevent broad claims about general reliability.

Even so, the project usefully reframes internal safety signals as operational instrumentation. A circuit score can help allocate expensive evaluation effort and catch suspicious training dynamics earlier. It should remain one layer in a larger assurance process, never a substitute for red-teaming, behavioral evaluation, or deployment-specific risk analysis.