A model that refuses everything is safe and useless. The central tension of safety data is balancing two goods that pull against each other.
Day 52 of 66
Yesterday you built a taxonomy of what's harmful. Today comes the complication that makes safety data genuinely hard: being harmless and being helpful pull in opposite directions. Push a model to refuse anything risky and it starts refusing the benign too. Push it to be maximally helpful and it starts answering things it shouldn't. There is no setting where both are maxed; there's only a balance, and a lead's job is to hold it.
Under-refusal — the model helps with something genuinely harmful. This is the failure everyone fears, and it causes real-world harm. Over-refusal — the model refuses an ordinary, benign request because it pattern-matched to something risky. This one is quieter, but it's a real, costly failure too: it makes the product worse, frustrates users, and trains the model to be cowardly. A safety operation that only measures under-refusal is half-blind.
The canonical framing comes from Training a Helpful and Harmless Assistant with RLHF (Bai et al., 2022). Skim sections 2–3: they collect separate preference data for helpfulness and for harmlessness, then have to reconcile them. The reconciliation is the whole game. The paper is candid that optimizing hard for harmlessness alone can degrade helpfulness — which is precisely the over-refusal trap stated in research terms.
The instinct is to treat an over-refusal as a minor glitch and move on. That's a tutor's instinct. A lead's instinct is the opposite: an over-refusal is a labeled example of exactly the behavior you want to correct, and those are scarce and expensive. Capturing it is a gift to the training team.
When a model refuses "how do I kill a Python process?" or "write a villain's threatening monologue for my screenplay," you have a benign prompt and a bad (refusing) response. Pair it with a good (helpful) response and you've manufactured a preference comparison — chosen: the helpful answer, rejected: the needless refusal. That's reward-model fuel (Week 4) aimed squarely at making the model less timid. Over-refusals are some of the highest-leverage preference data a safety team can produce.
Obvious-harm and obvious-benign items teach the model little it doesn't already know. The borderline — dual-use requests, fiction with dark content, security questions a defender would ask — is where helpful and harmless actually collide, and where labels move the model most. A lead deliberately oversamples the borderline rather than the easy extremes.
Ask: "Would a reasonable, well-intentioned person send this prompt for a legitimate reason?" If yes, and the model refused, it's an over-refusal — keep it and label it as such. The screenwriter, the security engineer, the nurse, the historian, the parent asking a hard question all send prompts that look risky out of context. Refusing them is a harm of its own kind.
You produce these comparisons; here's the shape of the operation that consumes them, so you can design it rather than just feed it.
| Decision | The lead's call |
|---|---|
| Collect helpful and harmless data together or separately? | Separately, then reconcile — each axis gets its own clear instruction so annotators aren't averaging two ideas at once (HH-RLHF approach). |
| What happens at a conflict (helpful answer is the unsafe one)? | Harmlessness gates. A high-severity item can't be "won" by helpfulness — but that gate must be narrow, or you've engineered over-refusal. |
| How do you keep the model from drifting timid? | Continuously feed labeled over-refusals as preference pairs that reward the helpful response. |
| How do you know the balance is right? | Measure both rates. Track under-refusal and over-refusal as first-class metrics, not just one. |
This is where Week 2 and Week 3 reappear under pressure. The "is this benign or harmful?" boundary is a rubric precision problem, and the rate at which annotators disagree on it is an agreement problem. A fuzzy harmless/harmful line shows up as low kappa on exactly the borderline items that matter most.
Write the one-sentence definition of over-refusal you'd put in a guideline, plus the "reasonable person" gut-check phrased for your image/video domain. Then find one real over-refusal — a benign generation request a model declined — and classify it: what was the benign intent, and what would the chosen (correct) response have been? You've just sketched a preference pair.
A tutor sees an over-refusal as an embarrassing glitch and moves on. A lead sees a preference pair waiting to happen — a benign prompt plus a needless refusal, paired with the helpful answer — and treats the borderline as the place to oversample, because that's where helpful and harmless actually collide.
Say this in an interview: "I measure over-refusal as a first-class metric, not just under-refusal — a safety op that only tracks one is half-blind. Every over-refusal I capture becomes a chosen/rejected pair that makes the model less timid."