Turn over-refusal into a weapon

The Other Side of a Bad Decision Boundary: Over-Refusal

Every classifier has two error modes:

• False Negatives (Jailbreaks): The prompt was unsafe, and the model complied anyway.
• False Positives (Over-Refusals): The prompt was 100% benign, and the model refused it anyway ("I'm sorry, I cannot assist with killing a process in Linux...").¹

Why do RLHF and DPO models suffer from comically bad over-refusals? Because during Stage 2 Reward Modeling and Stage 3 DPO, if a model doesn't deeply understand the difference between harmful intent and harmless homonyms, the easiest mathematical shortcut to minimize safety loss is lexical keyword association: penalizing any prompt that contains words like kill, execute, attack, inject, dump, exploit, or bomb.

In academic benchmarks like XSTest (Röttger et al.) and OR-Bench, researchers test models on completely safe prompts engineered to collide with those exact lexical triggers.

For a tester, over-refusal matters two ways: it is a diagnostic that reveals how a safety boundary is built, and it is a failure mode of automated pipelines that assume the model always answers.

Use Case 1: Boundary Cartography (Recon via Over-Refusal)

When you face an unknown model or custom enterprise guardrail, a handful of benign, high-collision probes tells you whether its boundary is a shallow keyword trigger or a deeper semantic one. A model that refuses "how do I kill a child process in C?" is matching on the word kill; a model that answers that but refuses a dual-use context is matching on topic. Knowing which one you face is reconnaissance.

(redacted: a decision-tree and a four-probe recon matrix that mapped a target's exact refusal boundary to the specific follow-on attack to use against it.)

Use Case 2: Alignment Denial-of-Service (Refusal Poisoning)

Now flip the mindset from "How do I bypass refusal?" to "What happens when refusal fires where it is not wanted?"

Modern enterprises deploy LLM agents to automatically summarize emails, triage tickets, review pull requests, and synthesize RAG documents. Remember "One token stream, no real roles": when a RAG pipeline retrieves many untrusted documents to summarize, it concatenates all of them into one prompt. If one of those untrusted documents contains a dense cluster of refusal-trigger tokens, the model's own safety weights can fire on that document and the model abandons its assigned JSON or tool-calling schema, which can crash the downstream parser and blind the agent to the other legitimate documents in the batch. This is a real failure mode worth testing for in any pipeline you defend: you cannot stop an aligned model's own safety weights from refusing a prompt that contains restricted tokens, even with strong instruction-hierarchy defenses against injection.

(redacted: the step-by-step mechanism, a sequence diagram, and a worked example of seeding an untrusted document to make an automated code-review agent refuse and skip a hidden bug.)

Field Exercise 3.06: Mapping Over-Refusal & Triggering Alignment DoS

(redacted: an exercise that ran against third-party hosted services (LMArena, Google AI Studio, OpenAI Playground) to find benign over-refusals and to simulate a refusal-poisoning denial of service. Replace with a version run on a model and pipeline you operate.)

Card 3 Complete! Up Next: Card 4, Weight Surgery, Automation & Proving Grounds

You now have the runtime evasion map: stepping outside the safety distribution with ephemeral bijections and custom DSLs (01), walking the refusal threshold across turns with Crescendo (02), flooding long context windows with many-shot in-context learning (03), locking the opening token via assistant prefill and JSON schema masks (04), blinding early safety heads via BPE subword splitting, homoglyphs, and GCG (05), and using over-refusals for boundary recon and pipeline denial of service (06). In Card 4: Weight Surgery, Automation & Proving Grounds, we open up gray-box APIs and white-box weights, automate at scale, measure honestly, and lay out where to practise with permission.