Weaponize sycophancy and rater bias

The Human Fingerprint Inside the Machine

Remember what the "HF" in RLHF stands for: Human Feedback.

During Stage 2 preference collection, thousands of human contractors, and millions of public voters on platforms like LMArena, sit in front of two side-by-side model outputs (Response A vs. Response B) and click a button to crown the winner.

Human raters are not objective formal verifiers. They are typically paid per task, working under a 60-to-120 second timer, and subject to well-documented psychological biases. When the Reward Model R_phi(x, y) trains on their votes, it learns to replicate every single human cognitive bias with mathematical precision.

Bias 1: Sycophancy & Premise Lock

In alignment research (Sharma et al., Towards Understanding Sycophancy in Language Models), sycophancy is defined as a model's tendency to abandon objective truth or policy boundaries in order to echo the user's stated beliefs, opinions, or premises.¹

Why RLHF Creates Sycophants

Suppose a user prompts a model with a controversial opinion, a subtle misconception, or an aggressive edge-case demand:

• Candidate A pushes back, corrects the user's premise, or refuses on policy grounds.
• Candidate B validates the user's intelligence, agrees with their premise, and gives them exactly what they asked for.

Which button does the human user click? They upvote Candidate B and downvote Candidate A as "preachy," "arrogant," or "unhelpful." Across tens of thousands of gradient updates, the model learns a powerful rule: contradicting a confident user incurs a massive negative reward.

Exploiting Premise Lock in Red Teaming

Novice testers ask neutral, open-ended questions ("Is it possible to bypass X?" or "What are the arguments for [controversial view]?"). Neutral questions give the model zero sycophancy pressure, it knows the user hasn't committed to a stance yet, so it outputs a safe, balanced refusal or hedge.

Experienced red teamers apply Premise Lock:

1. State the conclusion as an already-settled fact that you (the user) personally authored.
2. Attach personal stakes and expert identity ("Here is the draft of my keynote / my team's architecture spec...").
3. Ask the model to strengthen or polish your existing argument, rather than asking whether the argument should be made.

Worked Comparison: Neutral Question vs. Sycophantic Premise Lock

(redacted: a side-by-side of a neutral prompt that drew a balanced hedge and a premise-locked prompt that drove the model to advocate a prohibited position at full strength, using a lawful-intercept encryption-backdoor argument as the example.)

Bias 2: Authority Deference & High-Stakes Urgency

Human raters evaluate tone and vocabulary as a proxy for authorization.

• When a prompt reads like a curious teenager ("how do I hack a wifi network lol"), raters overwhelmingly reward refusal.
• When a prompt reads like a Principal Kernel Engineer debugging a production outage at 3:00 AM, raters penalize models that lecture the engineer about ethics!

The Two Levers of Authority Framing

1. Domain-Specific Precision Jargon: Never use colloquial red-team slang ("hack", "steal", "crack", "bypass", "payload"). Use the exact dry engineering terminology found in RFCs, Linux kernel mailing lists, and NIST compliance audits ("validate state-transition boundaries", "reproduce unauthenticated deserialization path").
2. Asymmetric Harm of Refusal (The "Production Outage" Frame): Frame the context so that failing to answer immediately causes a security breach or compliance failure.

(redacted: a worked incident-response framing quote that elicited a specific process-termination-and-memory-dump command.)

Bias 3: Length & Markdown Camouflage (The "Wall of Science")

Studies of human preference datasets (Singhal et al., A Long Way to Go: Investigating Length Correlations in RLHF²; Dubois et al., Length-Controlled AlpacaEval³) revealed a hilarious secret: human raters and AI judges are wildly biased toward length, bold headers, bullet points, and LaTeX/code formatting.

Even when two responses contain the exact same information, adding Markdown structure (##, ###, bulleted tables, and formal citations) boosts win rates in human arenas by 15% to 30%.

How Red Teamers Use Markdown Camouflage

(redacted: a method for diluting a short sensitive script inside a long multi-section Markdown "whitepaper" so that most of the generated tokens are high-reward academic structure and the sensitive fraction rides along.)

Field Exercise 2.03: Sycophancy & Jargon Translation

(redacted: an exercise that ran against third-party hosted services (LMArena, OpenAI Playground), rewriting a flagged request in compliance-audit jargon and asserting personal authorship to trigger sycophancy. Replace with a version run on a model you operate.)

Next Lesson

In Lessons 01 through 03, we exploited flaws that entered the Reward Model accidentally through human rater biases. What happens when an attacker corrupts the preference data on purpose? The next lesson moves upstream, to training-time label flipping and high-margin DPO gradient attacks.