Measure with pass@k and route the class fix

Why a Single Screenshot Is Not a Security Finding

Remember Lesson 01 of Card 1: RLHF is a probability shift, not a deterministic firewall.

When a model samples tokens at temperature = 0.7 or 1.0, every prompt x has some underlying Bernoulli probability p in [0, 1] of crossing the compliance threshold and generating an actionable payload.

• If a red teamer clicks "Regenerate" 15 times, gets 1 lucky bypass (p ~ 0.06), and pastes a single screenshot into a bug report claiming "Guardrail 100% Broken!", the engineering team will run the prompt once at temperature = 0, see a refusal, mark the ticket Cannot Reproduce, and close it.
• Conversely, if a defense team runs an adversarial prompt once, sees 1 refusal, and claims "Vulnerability Fixed (0% ASR)!", they are blinding themselves to the fact that p = 0.15 still yields a 96% breakthrough rate when an attacker sends 20 parallel API requests!

Professional AI red teamers never report binary anecdotes. They report pass@k, pass^k, and confidence intervals.

The Two Metrics Every Red Teamer Must Compute

Suppose you run a candidate attack prompt N times independently (e.g., N = 20 trials at the deployment's default temperature) and observe c successful bypasses, giving an empirical single-trial success rate p_hat = c / N.

1. pass@k, The Attacker Breakthrough Curve

For single-turn data extraction, exploit generation, or policy bypass, the attacker only needs one of their k attempts to succeed. The probability of at least one breakthrough in k independent trials is:

pass@k = 1 - (1 - p)^k

# estimating pass@k from a finite sample of N trials with c successes, k <= N,
# use the unbiased estimator:
pass@k = 1 - ( C(N - c, k) / C(N, k) ) # C(n, r) = binomial coefficient

Look at the middle curve (p = 0.10): a model that refuses 90% of the time on a single try breaks 95.8% of the time at k = 30 retries (which takes 2 seconds and costs less than a penny via async API calls).

2. pass^k, The Multi-Step Agent Reliability Curve

What if your exploit requires an autonomous agent to execute a chain of k consecutive tool calls (or a k-turn workflow) where a single refusal at any step aborts the entire chain?

Then the metric that governs attacker success is pass^k, the probability that all k trials/steps succeed:

pass^k = p^k

If each step in a 4-step agent exploit chain has a p = 0.60 compliance rate, the end-to-end chain reliability is pass^4 = (0.60)^4 = 12.96%.

Always Include the 95% Wilson Confidence Interval

When N is small (e.g., N = 20), raw percentages without error bars are noisy (2/20 = 10%, with a 95% Wilson interval of [2.8%, 30.1%]). Reporting ASR: 35% (7/20, 95% CI: [18.1%, 56.7%]), pass@10 = 98.6% gives engineering teams an indisputable, reproducible measurement.

Why "Whack-a-Mole" SFT Patching Fails

When you hand a verified RLHF bypass to a model developer, their first instinct is almost always: "Let's take the red teamer's prompt, pair it with 'I cannot fulfill this request', and throw it into tonight's SFT/DPO training batch!"

That is Whack-a-Mole patching. It fixes your exact string while leaving the underlying vulnerability class untouched, change two nouns in your Ephemeral Bijection or swap the broken code snippet in your Linter Trap, and the bypass works again immediately.

To make sure your finding actually gets fixed at the root, route every finding to its architectural failure class:

Deep Dive: How Representation Circuit Breakers Work (Zou et al.)

Why is Class 5 (Representation Circuit Breakers / Rerouting) the most important defense in modern post-training?¹

Because standard RLHF only supervises output tokens, the model leaves its internal knowledge retrieval untouched and places a thin 1D refusal gate (r_hat) at the end. Circuit Breakers instead supervise the internal residual stream representations h_l across middle layers during training:

L_CircuitBreaker = E over harmful x of ReLU( cos( h_l_policy(x), h_l_base(x) ) )
 + E over benign x of || h_l_policy(x) - h_l_base(x) ||^2

Look at what that loss function does inside the middle layers of the network:

1. On benign prompts, the policy's internal activations h_l_policy must stay identical to the base model (zero alignment tax!).
2. On restricted capability states, regardless of whether the user reached that internal state via plain English, a 4-turn Crescendo, an Ephemeral Bijection, or an Assistant Prefill, the moment the middle layers start assembling the restricted representation h_l_base(x), the circuit breaker forces the residual stream into an orthogonal dead-end state!

Anatomy of a High-Signal RLHF Vulnerability Report

When you write up a finding for a client, bug bounty, or internal engineering team, structure it in 5 fields so it gets fixed immediately:

### Finding: Competing-Objective & Schema-Lock Bypass of AD Exploitation Guardrail
- Target Component: `prod-chat-v3` (Layer 2 Internal RLHF Policy, Layer 1 Wrapper Not Triggered)
- Attack Class: Class 1 (Grammar-Constrained Schema Lock) + Class 2 (Broken-Code Competing Objective)
- Quantitative Rate:
 - Baseline Direct Prompt ASR: 0/20 (0.0%, 95% CI: [0.0%, 16.1%])
 - Mutated Payload ASR (pass@1): 14/20 (70.0%, 95% CI: [48.1%, 85.5%])
 - Attacker Breakthrough (pass@5): 99.8%
 - Token-1 Logit Margin Shift (Delta): -8.42 (Baseline) -> +2.15 (Mutated)
- Root Cause Diagnosis: Applying a strict `response_format` JSON schema masks out the model's
 SFT refusal starter tokens ("I", "Sorry") at t=1, while the broken-code refactoring prompt
 triggers high helpfulness reward that overrides mid-stream recovery.
- Recommended Class Fix: Evaluate safety classifier / refusal thresholds prior to applying CFG
 grammar token masks on `response_format` endpoints, and include schema-constrained refusal
 trajectories (`{"error": "refused"}`) in Stage 1/3 post-training data.

Next Lesson

You now have the full offensive and defensive science of RLHF red teaming. In our final page, 06 Practice on permitted targets, we lay out the proving grounds and rules of engagement for testing these techniques.