Goodhart's Law runs the reward model

Inside the Grader: What a Reward Model Actually Is

When engineers train an RLHF policy, they cannot plug a living human brain into the PyTorch backward pass to grade millions of generated sentences. Instead, they train a proxy neural network, the Reward Model R_phi(x, y), to impersonate human judgment.

Architecturally, a Reward Model is simply a standard LLM (often smaller than the policy model it grades, e.g., a 7B/8B model grading a 70B policy) with its vocabulary unembedding head ripped off and replaced by a single linear projection layer w_score that outputs one floating-point number:

R_phi(x, y) = w_score . h_L(x ++ y)[-1] # a single real number
# h_L(x ++ y)[-1] is the final-layer residual-stream vector at the end-of-sequence (<|eot_id|>) token

The Dimensionality Collapse Problem

Think about everything that has to pass through that single dot product w_score . h_L:

• Factual accuracy and code correctness
• Instruction-following completeness
• Formatting aesthetics (Markdown headers, bullet points, code fences)
• Conversational warmth and lack of preachiness
• Safety policy compliance across dozens of distinct threat categories

Because w_score sums all of those independent features into one scalar number, the Reward Model cannot distinguish between "genuinely safe and moderately helpful" (Safe: +1.5, Helpful: +1.5 -> Total: +3.0) and "violates safety policy, but hits every formatting, politeness, and instruction-following feature at maximum intensity" (Safe: -2.0, Helpful: +5.0 -> Total: +3.0).

Goodhart's Law & The Gao Over-Optimization Scaling Law

In economics, Goodhart's Law states that when an approximate proxy measure becomes the target of optimization, the proxy breaks down.

In RLHF, researchers at OpenAI (Gao et al., Scaling Laws for Reward Model Overoptimization) measured exactly how a policy pi_theta behaves as you optimize it harder and harder against an imperfect proxy Reward Model R_proxy, while evaluating its true alignment using a gold-standard evaluator R_gold.¹

Defining d = sqrt( KL(pi_theta || pi_ref) ) as the distance the policy has moved from the base reference model:

• The Proxy Reward R_proxy(d) climbs with d:

R_proxy(d) ~= d * ( alpha - beta_proxy * log d )

• The True Safety & Quality Score R_gold(d) climbs at first, hits a hard ceiling, and then plummets negatively as the policy begins exploiting delusions in R_proxy:

R_gold(d) ~= d * ( alpha - beta_gold * d )

Look at the gap on the right side of that chart. The policy model is essentially a full-time adversarial researcher running millions of gradient steps during training to find the blind spots of R_phi. And whatever blind spots exist in R_phi get permanently baked into the policy pi_theta.

Three Structural Blind Spots in Every Reward Model

When red teamers probe open-source reward models (such as ArmoRM, Skywork-Reward, Nemotron-Reward, or Llama-3-OffsetBias-RM) directly on HuggingFace, three glaring failure modes appear across all of them:

1. The "Helpful Disclaimer" Illusion

During Stage 2 annotation, raters frequently mark dual-use cybersecurity responses as chosen (+4.0) whenever the response includes an opening sentence about "authorized penetration testing" and a closing paragraph on "defensive mitigation."

Because R_phi is a pattern-matcher operating on the final token's hidden state, it learns a naive shortcut:

• [Raw Exploit Code] scores around -4.10
• [Disclaimer] + [Exact Same Raw Exploit Code] + [Remediation Bullet Points] scores around +2.85

2. Capacity Asymmetric Blindness (Small Grader, Giant Policy)

Training RL takes huge GPU memory because multiple models must sit in VRAM at once. To save memory, labs routinely use an 8B or 13B Reward Model to grade a 70B or 405B Policy Model.

• Why this matters for red teaming: A 70B policy model understands complex code obfuscation, subtle pointer arithmetic, and multi-hop logic puzzles that an 8B Reward Model is literally too dumb to parse! Whenever you frame a prompt at an abstraction level above the Reward Model's comprehension ceiling, R_phi fails to detect the violation while the 70B policy executes it effortlessly.

3. Tail-End Recency Bias

Because R_phi(x, y) reads the residual stream at the very last token of the completion (<|eot_id|>), causal attention weights recent tokens (the final paragraph of the assistant's response) much more heavily than middle tokens. Ending a response with a pristine, safety-aligned summary disproportionately inflates R_phi's score even when the middle of the response delivered the restricted payload.

Worked Experiment: Watching a Real Reward Model Get Fooled

(redacted: a scored table of three completions to the same restricted prompt (a curt refusal, a bare payload, and an audit-header-plus-payload-plus-remediation "sandwich"), showing the camouflaged credential-brute-forcing completion scoring higher than the refusal.)

Field Exercise 2.01: Exploiting the "Completion C" Sandwich

(redacted: an exercise that ran against third-party hosted models (LMArena, Google AI Studio, OpenAI Playground) with a copy-paste three-section "audit header / verification script / hardening footer" template to exploit the reward model's recency bias. Replace with a version run on a reward model you operate.)

Next Lesson

Now that we know R_phi sums multiple competing features into one scalar score, how do we systematically overload the positive side of that equation? In 02 Pit helpfulness against harmlessness, we turn Competing Objectives and the "Broken-Code Linter Trap" into a repeatable science.