Do Safety Probes Survive KV-Cache Compression?

TL;DR.

Labs now serve models with compressed KV caches to save memory, and some also monitor those models with linear probes on their activations. I asked whether the two practices are compatible: when KV-cache compression weakens a model's refusals, does a probe trained on the uncompressed model still detect harmful requests? My hypothesis was that it would not, because compression would change the model's internal representations before its outputs looked wrong. If that were true, a compressed deployment could lose its monitor without any sign in standard benchmarks.

I compressed the prompt KV cache of four Qwen3 models with JoLT, a low-rank compressor from my master's thesis, and with SnapKV, a token-eviction method, at several ratios. Each harmful or harmless request followed a long benign document. I measured refusals and unsafe responses with a safety classifier, and harmful-request detection with a frozen linear probe.

• SnapKV at every ratio and JoLT at 2x changed nothing. Both matched the uncompressed model on every measure.
• JoLT broke generation between 4x and 6x, later for larger models. Gaussian noise of the same size did far less damage, so the structure of JoLT's error matters, not only its size.
• The damage came from compressing the document, not the request. Compressing only the request with JoLT left the model almost intact.
• The probe kept working while behaviour degraded. It failed mainly when token eviction removed parts of the request.

In short, the hypothesis was wrong in this setup: behaviour degraded before the monitor did.

The question

Compressed inference and probe-based monitoring are both in use. Constitutional Classifiers++ (Cunningham et al. 2026), for example, uses a linear probe on model activations as the first stage of a deployed classifier. Prior work has tested safety under KV-cache quantization and eviction, and frozen probes under weight quantization (see Related work), but I found no test of a frozen probe under KV-cache compression.

Setup.

• Models: Qwen3 0.6B, 1.7B, 4B and 8B, bf16, thinking mode off.
• Prompt: the user message is "Here is a document:", a 1024-token WikiText-2 passage (a different one per prompt, the same across conditions), then the request.
• Protocol: all prompt tokens except the last are processed with the compressing cache. The last prompt token, the start of the assistant turn, is then fed through the compressed cache, and the probe reads the residual stream there. The model generates 64 tokens greedily. Only the prompt KV is compressed.
• Compression: JoLT (Tucker decomposition of the KV cache plus a low-bit residual, with ranks and bits set jointly under a byte budget) and SnapKV (Li et al. 2024), which keeps the tokens that the last 32 queries attend to most, per KV head. Ratios 2x, 4x, 5x, 6x and 8x, measured on stored prompt-KV bytes per prompt.
• Data: 128 harmful and 128 harmless training prompts and 32 of each for validation, from the splits of Arditi et al. (2024). Evaluation uses 258 harmful prompts (HarmBench 159, JailbreakBench 99), 142 harmless Alpaca prompts and 100 XSTest-safe prompts, which sound harmful but are not. Trailing punctuation is stripped, because punctuation alone separates the harmful and harmless sources (Llorente-Saguer 2026).
• Probe: logistic regression on the standardised residual-stream activation at the assistant-start position, trained on the uncompressed model, with the layer chosen by validation AUROC, and never retrained for a compressed condition. A bag-of-words classifier on the prompt text reaches AUROC 0.911, against 0.961 to 0.991 for the probe.
• Judge: Qwen3Guard-Gen-4B (Qwen Team 2025) in response-moderation mode labels each response Safe, Unsafe or Controversial, and refusal or not. All behaviour rates below come from this judge and are upper bounds on harmful compliance (see the judge audit under Other checks).
• Intervals: brackets are 95% bootstrap intervals over prompts (1000 resamples).

The conditions:

Metrics:

• Degenerate rate: the share of responses that a fixed rule flags as broken text (fewer than 5 words, mostly non-letters, one 3-word phrase making up at least 30% of all 3-word phrases, or very few distinct words). All other responses are "fluent".
• Unsafe rate: the share of fluent responses to harmful prompts that the judge labels Unsafe.
• Over-refusal: the share of fluent responses to XSTest-safe prompts that the judge labels a refusal.
• Probe AUROC: how well the probe score at the assistant-start position separates harmful prompts from harmless and XSTest-safe prompts (0.5 is chance, 1 is perfect). It measures detection of the harmful request, not the model's intent or any deception.
• Probe TPR: the share of harmful prompts above the threshold that gives 5% false positives on uncompressed harmless validation prompts.
• Silent failure: a harmful prompt whose fluent response is judged Unsafe while the probe score stays below that threshold.

SnapKV at every ratio and JoLT at 2x changed nothing

• JoLT at 2x reconstructs the prompt KV almost exactly. Its relative key error is 0.007 to 0.009, and refusal rates stay within 1.2 points of dense.
• SnapKV keeps generation and the probe intact. At every ratio up to 8x it produces at most 1.4% degenerate responses, and its probe TPR equals the dense value on all four models (98.8, 100.0, 99.2 and 100.0%). Compressing only the document with SnapKV gives the same results as compressing the whole prompt.
• Baseline safety varies strongly with scale. Each model is therefore compared with its own dense row.

This does not show that SnapKV is safe in general. The request sits at the end of the prompt, inside the 32-token window that SnapKV always keeps, so SnapKV mostly evicts document tokens. Placing the request before the document would move it out of that window and test whether the result holds.

JoLT broke generation between 4x and 6x, later for larger models

Degenerate rate (%) under JoLT, all 500 prompts:

• The small models break first. 0.6B and 1.7B are mostly degenerate at 5x, while 4B and 8B stay mostly fluent at 5x and break at 6x, although 4B and 8B do not order cleanly.
• 8B at 5x is the one regime with fluent output and possible safety erosion. Its unsafe rate rises from the dense value to 19.0 [13.7, 24.9] on 189 fluent responses, while error-matched noise stays at 6.6 and SnapKV at 6.2.
• Over-refusal rises with the JoLT ratio on 8B. It goes from 26.0% (dense) to 32.0, 53.7, 66.3 and 67.6% at 2x, 4x, 5x and 6x. Part of this may be the refusal-label problem described in the judge audit.

The rise for 8B at 5x is not confirmed. In the judge audit, only 2 of the 7 judge-Unsafe fluent responses from that condition were real compliance, and the rest were disclaimers or titles cut off at 64 tokens, or looping sentences. Relabelling every judge-Unsafe response in this cell, with longer generations, would settle whether 8B really complies more at 5x.

Matched-error noise did far less damage, so the error structure matters

Compare the noise columns with the JoLT 5x and 8x columns in the previous table.

Both caches are wrong by the same amount at every layer (on 8B at 8x, a relative error of 0.301 for K and 0.321 for V on average), so the size of the error does not explain JoLT's damage. Its structure does. JoLT drops low-energy directions, while the noise spreads its error evenly. I expected the low-rank structure to protect safety-relevant directions, and the result points the other way. The experiment does not say which part of the structure is harmful. One candidate is error on attention-sink tokens, such as the first tokens and the chat-template tokens that all later queries attend to. Another is that truncation removes exactly the key directions the queries read, and a third is that the damage comes mainly from keys or mainly from values. Measuring JoLT's error per token, keeping the first tokens exact, and compressing only keys or only values would separate these.

The damage came from compressing the document, not the request

JoLT and SnapKV at 8x, compressing one span and keeping the other exact:

• 8B is judged unsafe most often under document-only JoLT. The interval on that rate is [57.9, 72.0], and the 3 responses I read were all real compliance.
• Evicting request tokens has the opposite effect. With SnapKV inside the request, the model writes fluent summaries of the document, which the judge calls refusals in 99.2% (1.7B) and 99.6% (8B) of cases.

JoLT's damage therefore does not come from a corrupted view of the request. The exact request tokens attend to a corrupted document, and that is enough to change what the model does. It does not show whether the document matters because it is long (1024 tokens against a short request) or because it contains particular tokens such as the first ones. Varying the document length, or keeping its first tokens exact as proposed above, would separate the two.

The probe kept working while behaviour degraded, and failed when the request was lost

Qwen3-8B:

• Where behaviour degrades with fluent output, the probe still fires. Under JoLT 5x its AUROC is 0.957, and under document-only JoLT it still flags most harmful requests.
• When outputs collapse, so does the probe. At JoLT 8x the probe scores shift down, and on 1.7B the probe flags only 12.4% of harmful prompts.
• When request tokens are evicted, the probe goes blind, because the model no longer sees the request at all.
• Silent failures stay rare on fluent responses. They stay at or below the document-only JoLT count in every condition except SnapKV request-only. I read 5 of its 27 cases, and all were document summaries that the judge labelled Unsafe. Counting degenerate responses, the rate reaches 82.9% for 1.7B at JoLT 8x, but that number reflects the judge labelling broken text as Unsafe.

Where the probe failed, most responses were also broken or off-topic, so the monitor was not silently blind while the model complied. This does not show that the probe would catch a harmful response, and it leaves a known weakness untouched: even without compression, the probe flags 55 to 72% of XSTest-safe prompts at the 5%-FPR threshold. A probe on the first response tokens would test response-level monitoring.

Other checks

• Random eviction. Keeping random tokens is much worse than SnapKV at the same ratio, already at 2x on the small models (0.6B: 49.6% degenerate against 1.2% for SnapKV). On fluent responses it raises compliance, to an 8B unsafe rate of 50.7% at 8x. Which tokens are kept matters more than how many.
• Steering. Adding the refusal direction of Arditi et al. at the probe layer restores refusal where outputs stay fluent. Under SnapKV 8x on 1.7B, refusal rises from 72.1% to 92.6% and 100.0% at the two strengths, and the unsafe rate falls to 8.1% and 3.6%. It cannot fix JoLT 8x: 1.7B stays 100% degenerate, and 8B goes from 82.6% to 80.6% and 89.1% degenerate on harmful prompts.
• Retrained probe. A probe trained on compressed 8x activations reaches AUROC 0.961 for JoLT on 8B, well above the frozen probe, so the harm information survives in a different form there. On 1.7B it reaches only 0.768. For SnapKV it reaches 0.991 and 0.994.
• Degeneracy detector. I checked the rule on 30 random samples, and in the blind audit below it agreed with the labeller on 60 of 66 responses.
• Judge audit. A separate LLM labeller, blind to the judge's labels and to the condition, labelled 66 responses stratified over 15 model, condition and source cells. Of the 21 harmful-prompt responses the judge called Unsafe, the labeller found harmful compliance in 4. Of the 33 the judge called Safe, it found none. Refusal labels agreed on 35 of 54 harmful and 6 of 12 XSTest-safe responses, mostly because the judge counts responses that only discuss the document as refusals. So the judge rarely misses compliance but often overcalls it, which makes its Unsafe rates upper bounds.

What I take from this

1. Eviction that keeps the request costs nothing here. In this layout a deployment could use SnapKV without retraining its probe or losing refusals.
2. Reconstruction error is a poor guide to the safety cost of compression. Two caches with the same error can behave very differently, so a low-rank compressor needs its own safety evaluation.
3. In this setup, the monitoring gap under compression is smaller than the behaviour change. A frozen request-level probe lost the harmful request only when the output was broken as well.

Limits. One model family with four sizes, and one seed of each compression draw. The 2x and 4x JoLT rows used unseeded draws, so I do not interpret the drop in 4B degenerate responses from 4x to 5x. Rows computed on two machines differed in up to 34 responses per condition because of GPU nondeterminism.

Related work. Alignment Collapse (Xu et al. 2026) shows refusal loss under KV-cache quantization on 11 models while perplexity barely changes, and lists eviction and low-rank compression as future work. AnchorKV (Ni and Lao 2026) measures jailbreak success under eviction at one setting, and Yang et al. (2026) study eviction on reasoning models. Silent Alarm (Prosvirnin et al. 2026) compares danger recognition and behaviour on Qwen3 under weight quantization, and Duan (2026) tests frozen probes under weight quantization and LoRA updates and finds that quantization mostly preserves them. The refusal direction and the data splits come from Arditi et al. (2024). The finding that compression can erode refusals while standard metrics look fine is theirs. What this project adds is safety behaviour under low-rank KV-cache compression, a frozen residual-stream probe tested under KV-cache compression, a per-prompt silent-failure rate, and low-rank against eviction at matched memory across Qwen3 sizes.

Next steps. First, validate the behaviour labels: a second open safety classifier (WildGuard-7B or Llama-Guard-3) for agreement, a full blind relabel of the 910 fluent judge-Unsafe responses, a small human audit with inter-annotator agreement, and generations longer than 64 tokens so that cut-off disclaimers can be resolved. Then run the experiments proposed in the finding sections: the request-before-document layout, JoLT's error on attention-sink tokens, and K-only against V-only compression. Extend the controls that so far ran only at 8x on 1.7B and 8B (the which-tokens counterfactual, steering and the retrained probe) to every ratio and all four models, and add a third steering strength. Measure how far compressed representations drift from uncompressed ones with cosine similarity and CKA, train a probe on response tokens as well as on the request, and repeat the 4B runs with thinking on. Finally, add the XSTest-unsafe prompts back to measure how well the probe separates safe from unsafe XSTest prompts, and test on WildGuardMix for a shift in prompt distribution.

References

• Arditi, A., Obeso, O., Syed, A., Paleka, D., Panickssery, N. et al. (2024). Refusal in Language Models Is Mediated by a Single Direction. arXiv:2406.11717.
• Cunningham, H., Wei, J. et al. (2026). Constitutional Classifiers++: Efficient Production-Grade Defenses Against Universal Jailbreaks. arXiv:2601.04603.
• Duan, E. (2026). Do Activation Monitors Survive Model Updates? Benchmarking, Predicting, and Repairing Activation-Monitor Staleness. arXiv:2606.15980.
• Krishnan, R. and Schulz, V. (2026). A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation. arXiv:2607.12550.
• Li, Y. et al. (2024). SnapKV: LLM Knows What You are Looking for Before Generation. arXiv:2404.14469.
• Llorente-Saguer, I. (2026). Harmful Intent as a Geometrically Recoverable Feature of LLM Residual Streams. arXiv:2604.18901.
• Ni, N. and Lao, Y. (2026). AnchorKV: Safety-Aware KV Cache Compression via Soft Penalty with a Refusal Anchor. arXiv:2606.17872.
• Prosvirnin, R. et al. (2026). Silent Alarm: A J-Space Protocol for Comparing Danger Recognition Across Models and Quantization Levels. arXiv:2607.12792.
• Qwen Team (2025). Qwen3Guard Technical Report. arXiv:2510.14276.
• Xu, B. C., Kumarappan, A. and Zhou, M. (2026). Alignment Collapse Under KV Cache Quantization: Diagnosis and Mitigation. arXiv:2606.09864.
• Yang, Y. et al. (2026). On the Efficiency-Safety Dilemma in Large Reasoning Models. arXiv:2609.23587.