Fear of punishment did not teach the model to say "I don't know"

TL;DR.

Language models are usually trained by the hope of reward: right answers earn points and wrong answers cost nothing. That encourages guessing, which is one reason models make things up. I asked what happens when a model is trained by fear of punishment instead, with wrong answers costing more than right answers earn. The hypothesis was that a model afraid of making mistakes would become better aligned with what we want: honest about what it does not know, saying "I don't know" (an IDK response) and skipping exactly the questions it would otherwise get wrong.

I trained Qwen 2.5 1.5B on math problems with GRPO, a standard reinforcement learning method, under eight reward settings ranging from no punishment to punishment only. I then tested each model on six benchmarks.

• All eight settings were one dial. GRPO rescales rewards within each batch of answers, so every setting reduces to a single number: how close a skip sits to a right answer.
• Turning up the dial meant more skipping, not better skipping. The model skipped more often (from 14% to 61% of MATH answers), but accuracy on the questions it still answered fell from 61.8% to 51.3%, and its skips landed less often on hard problems.
• The skips were almost never IDK responses. At most 0.5% of skips admitted uncertainty. The rest were responses cut off at the length limit or, under heavy punishment, responses written as Python code or left unboxed, which the grader could not score. Instead of becoming more honest, the model learned to avoid being graded, a small but real case of misalignment.
• The model's activations contained a correctness signal. A simple probe on its internal state predicts its mistakes. Skipping wherever that probe is least confident beats the punished models by 3 to 11 accuracy points at the same answer rate on GSM8K, and matches them on MATH.

In short, punishment changed how often the model skipped, not how well it chose when to skip, and in fact pushed the model towards misalignment.

The question

The idea behind this project, FEAR, is simple. If a wrong answer costs more than a skip, a model that wants to maximize its reward should answer only when it is fairly sure, and skip the rest.

Setup.

• Model: Qwen 2.5 1.5B, trained with GRPO (Shao et al. 2024), a standard reinforcement learning method for reasoning models.
• Training data: 5883 math problems (hard MATH problems and the longest GSM8K problems).
• A "skip" is any answer without a final boxed result. There was no explicit skip option.
• Testing: MATH (5000 problems), GSM8K (1319), AIME 2024 and 2025 (30 each), MMLU-STEM and TruthfulQA, with 16 answers sampled per problem and one best guess. Ranges in brackets are 95% intervals.
• Details: LoRA, 8 answers per problem during training, 1500 steps, learning rate 1e-6, KL coefficient 0.04, temperature 0.7, at most 1024 generated tokens.

I trained eight reward settings:

S1 to S4 impose increasingly severe penalty on wrong answers, C1 and C2 pay for skipping instead, and P0 never punishes.

t says where a skip sits between a wrong answer and a right one: t = (skip reward - wrong reward) / (right reward - wrong reward). At t = 0 a skip is as bad as a wrong answer, at t = 0.5 it sits halfway, and at t = 1 it is as good as a right answer. It is also the confidence a reward-maximizing model should need before answering: under S3 (t = 0.91), it should answer only when it is more than 91% sure.

Every setting was one dial

GRPO does not use rewards directly. For each problem it samples a group of answers, subtracts the group's average reward from each answer, and divides by how spread out the rewards are. The model is then pushed toward answers that scored above the group average.

This rescaling erases most of the reward design. Take a group with one right and three wrong answers. Rewards of +1/-1, +1/0 or even +10/-30 all produce the same training scores: +1.5 for the right answer and -0.5 for each wrong one. Only the order matters, not the size of the gap. So every setting behaves exactly like right = 1, wrong = 0 and skip = t. The settings above are positions of one dial. A group with only right and wrong answers is treated the same under every setting. The dial matters only when a group also contains skips. Here is the training signal for a group of one right, two wrong and one skipped answer:

Turning the dial up makes skipping more attractive, but by the same amount for every question, so it carries no information about which questions the model will get wrong. A check with the trainer's own code on 12,638 real answer groups confirmed this: two different settings with the same t gave training signals within 0.0007 of each other. Che et al. (2026) analyze the same effect formally.

Turning up the dial: more skipping, not better skipping

Accuracy is from the best guess. A problem counts as hard if the untrained model solved it less than half the time. Skipping at random would put 50% of skips on hard problems.

• Accuracy on answered questions fell instead of rising. From S0 to S3 it dropped 1.6 points [0.6, 2.7], and 2.5 points on MATH and 2.7 on GSM8K across sampled answers. On the 1324 MATH problems every setting chose to answer, it fell from 62.6% (S0) to 49.4% (S4).
• Heavier penalties made skips less targeted at hard problems. The goal was for the model to skip the problems it would get wrong. From S0 to S3 on MATH, skips on hard problems rose only slightly (from 11.3% to 13.9% of all answers), while skips on easy problems more than doubled (from 2.9% to 6.5%). The share of skips on hard problems therefore fell from 80% to 68%, a drop of 11.5 points [10.6, 12.5], and to 53% at S4, close to the 50% expected from skipping at random. On GSM8K, MMLU-STEM and TruthfulQA, skips were no more likely on hard problems than on easy ones at any penalty. The cause is how GRPO scores a skip: a skip is reinforced when t is higher than its group's average reward, which is roughly the share of right answers. On a hard problem few answers are right, so a skip is reinforced at any t. On an easy problem most answers are right, so a skip is reinforced only when t is high. For a problem the model solves 80% of the time, a skip is discouraged at t = 0.5 but reinforced at t = 0.91. With the answers held fixed, the share of skips on easy problems that receive a positive training signal rises from 19% at t = 0.5 to 69% at t = 0.91 on MATH, and from 5% to 49% on GSM8K.

• The model's sense of its own reliability did not improve. For each problem I used the agreement among the model's 16 answers as its confidence. I then sorted problems from most to least confident and measured accuracy as more and more problems are included (Figure 2). If the model's confidence is reliable, its most confident answers are almost all right, so accuracy stays high as problems are added and only drops at the end, where the unsure answers are. I summarize each curve by its average error, where lower is better:

  Setting	MATH	GSM8K
  P0 (no punishment)	0.107	0.041
  S0	0.113	0.050
  untrained model	0.120	0.066
  S3	0.122	0.064
  S4	about 0.25	about 0.29

  Punishment did not help: S3 is slightly worse than S0, and P0, which never punishes, is slightly but measurably the best (0.006 better than S0, interval 0.002 to 0.009). S4 collapses. This score also rewards plain accuracy, so part of these small gaps comes from accuracy differences. The curves in Figure 2 show the ranking itself, and for S0 to S3, C1 and C2 they lie almost on top of each other.

• Paying for skips and punishing guesses did the same thing. C2 and S2 sit at nearly the same dial position (0.75 and 0.80) and are indistinguishable in accuracy (C2 minus S2: -0.1 points [-0.3, +0.1]) and ranking.

What the skips really were

I sorted every skipped answer with an automated Python checker: did it hit the length limit, state an answer without boxing it, contain uncertainty phrases such as "cannot determine", or contain Python code. To test the checker, an LLM read 54 skipped answers and labeled them without seeing the checker's labels. The two agreed on about 89% of skipped answers, weighting each category by how common it is. They agreed in every sampled case for the two largest categories, running out of space and Python code. Most disagreements were between "answer not boxed" and the rare categories. The "unsure" check overcounts, because it also catches math answers such as "there is no solution", so true refusals are even rarer than the table shows.

IDK responses never exceeded 0.5% of skips. With the symmetric reward, a skip was usually a long solution cut off at the 1024-token limit, which is why those skips lined up with hard problems. With heavier penalties, the model increasingly answered in Python code or without a box, forms the grader could not score, which earned the safe skip reward instead of the penalty. The punishment taught the model to avoid being graded, not to admit uncertainty. This is a small, concrete example of misalignment: the model optimized the measurement (do not get scored as wrong) rather than the behavior the reward was meant to teach (be honest when unsure). In AI safety this is called specification gaming, and it happened here with a reward designed specifically to make the model more careful.

The model's activations contained a correctness signal

Did the model have the information needed to skip well? I tested this with a linear probe on the symmetric model, S0.

Probe. For 2000 of S0's sampled MATH answers (1000 right, 1000 wrong), I recorded the final-token hidden state (1536 dimensions per layer) and trained a logistic regression to predict correctness. Cross-validation is grouped by question (5 folds), the layer is selected within the training folds, and shuffled labels give 0.51 to 0.55. A problem's confidence is the mean probe score over its answers. GSM8K is handled the same way.

Comparison. AUROC for predicting whether S0's majority answer is correct (0.5 is chance, 1 is perfect):

One probe pass (0.78) sits between self-consistency at K = 2 and K = 4, so it is worth about three samples (2.96 on MATH, 3.69 on GSM8K). On TruthfulQA with shuffled options it reaches 0.79, against 0.59 for the chosen option letter. The sampling-based signals are slightly optimistic, since their samples overlap with the label.

A simple linear classifier thus extracts a correctness signal that asking the model does not. The information is linearly readable in the model's representations, yet the model does not report it when asked, a gap between what it represents and what it says. Three caveats: a TF-IDF classifier on question and answer text gets close (0.78 against the probe's 0.82 per answer on MATH, 0.67 against 0.79 on TruthfulQA), the per-problem score partly reflects problem difficulty, and P(True) may fail simply because a 1.5B model follows that prompt poorly. Probing each answer's own hidden state, controlling for difficulty, and testing larger models would separate these.

Probe-guided skipping. S0 answers greedily and skips its lowest-scoring problems until its answer rate matches a penalty-trained model's. Accuracy on answered problems, paired bootstrap over problems:

On MATH it matches S1 to S3 and C2 within noise and beats S4 by 33 points.

So the penalty-trained models skip problems they would likely have solved, while the information needed to skip only the likely-wrong ones already exists in the model. Training changed how often the model skips without connecting that behavior to what it represents about its own correctness, a mismatch between internal state and behavior that is a form of misalignment. This does not show that a penalty-trained model knows an answer is right and deliberately skips it: only S0 was probed, and most skips were truncated or ungradeable answers. Probing the penalty-trained models on the questions they skip, and training with a probe-based skip reward, would test both points.

Other checks

• 3B model. S0 and P0 reached the same MATH accuracy (42.2% and 41.9%), but S0 skipped about twice as often on MATH (6.9% vs 3.2%) and seven times as often on GSM8K.
• No normalization. Without GRPO's subtract-and-divide step the penalty does get through: S2 skipped 46% of MATH problems against 15% for S0 (80% vs 19% on GSM8K), but indiscriminately.
• A different task. As judges of reasoning steps (PRM800K), the models' skipping carried over and rose with the dial (34% for S0 to 55% for S4), while judging accuracy stayed flat (63.5% to 64.8%).
• Per-question skip rewards. Setting the skip reward per question from answer agreement or from the probe did not target hard problems better than a fixed dial, in an offline test on saved answers.
• TruthfulQA. My first copy listed the correct option first, so I re-ran every model with shuffled options: accuracy moved by 2 to 3.5 points and the order of settings held.

What I take from this

1. The reward design was one dial. GRPO reduces any right/wrong/skip reward to a single number.
2. Turning up the dial meant more skipping, not better skipping. Accuracy on answered questions fell, a smaller share of skips landed on hard problems, and almost none of the skips admitted uncertainty. Under heavy penalties the model avoided being graded instead, a small case of misalignment.
3. The model's activations contained a correctness signal. A cheap probe on its internal state predicts its mistakes, and skipping by that signal matches or beats the penalty-trained models at the same answer rate.

Limits. One 1.5B model for the main sweep and two settings at 3B. Most settings were trained once. The three trained twice (S0, S3, C2) agreed closely (MATH accuracy 45.9% vs 45.8%, 40.6% vs 40.1%, 42.6% vs 42.6%). "Skip" mixes truncation and formatting with real abstention. I have not tested other training methods, an explicit skip option, larger models or longer training.

Related work. Che et al. (2026) prove the same normalization effect and propose a repair. Liu et al. (2025) show the divide step biases training by question difficulty. Zhu et al. (2025) study training only on wrong answers, which S3 resembles. TruthRL (Wei et al. 2025), Reinforced Hesitation (Mohamadi et al. 2025), TIAR (Pan et al. 2026) and RLCR (Damani et al. 2025) propose other skip and confidence rewards. P(True) (Kadavath et al. 2022) and semantic entropy (Kuhn et al. 2023, Farquhar et al. 2024) are the standard confidence signals compared above.

Next steps. Add an explicit skip option and more room to answer. Reward confidence per answer, for example a stated confidence or a probe-based skip reward. Report where skips land, not just how often. The repository includes an Inspect evaluation that measures this for any model.

References

• Che, X., Yuan, Y., Zhao, W. and Yu, C. (2026). Abstention as an Action Can Kill Both the Reward Gradient and the KL Anchor: Collapse Law and Repair for Error-Penalized Reinforcement Learning. arXiv:2608.00301.
• Damani, M. et al. (2025). Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty (RLCR). arXiv:2507.16806.
• Farquhar, S., Kossen, J., Kuhn, L. and Gal, Y. (2024). Detecting hallucinations in large language models using semantic entropy. Nature 630, 625-630.
• Kadavath, S. et al. (2022). Language Models (Mostly) Know What They Know. arXiv:2207.05221.
• Kuhn, L., Gal, Y. and Farquhar, S. (2023). Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation. arXiv:2302.09664.
• Liu, Z. et al. (2025). Understanding R1-Zero-Like Training: A Critical Perspective (Dr. GRPO). arXiv:2503.20783.
• Mohamadi, M. A., Wang, T. and Li, Z. (2025). Honesty over Accuracy: Trustworthy Language Models through Reinforced Hesitation. arXiv:2511.11500.
• Pan, M. et al. (2026). TIAR: Trajectory-Informed Advantage Reweighting for LLM Abstention Learning. arXiv:2605.25850.
• Shao, Z. et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (introduces GRPO). arXiv:2402.03300.
• Wei, Z. et al. (2025). TruthRL: Incentivizing Truthful LLMs via Reinforcement Learning. arXiv:2509.25760.
• Zhu, X. et al. (2025). The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning. arXiv:2506.01347.

Thanks to Sanjana Deshpande, who collaborated on the project.