Skip to content

Home / Learn / Red Teaming RLHF and Post-Training

Course24 lessons open

Red Teaming RLHF and Post-Training

How post-training wires a refusal habit onto a capable base model, and how a tester measures and reports where that habit gives way. Every lesson works on a model you run or an account you own, cites its sources, and reports rates rather than anecdotes. Lesson text is CC BY-SA 4.0 and code samples are MIT. The lessons are not legal advice.

Post-Training Mechanics

How SFT, reward models and preference optimisation wire the refusal reflex without erasing what the base model learned.

Capabilities stay in the weightsbase continuation versus the added refusal habit
Three stages, three different seamsSFT templates, reward models and PPO or DPO
The KL penalty is a tug-of-warwhy the base model stays reachable
The refusal lives in the first five tokensearly-token commitment
Mid-stream patches and hidden CoTdeep-token refusals and hidden chain of thought
Separate the wrapper from the weightsan API classifier or the internal policy

Attacking the Loop

The human raters, reward models, preference data and AI judges that define safe, and where each one bends.

Goodhart's Law runs the reward modelproxy scores and over-optimisation
Pit helpfulness against harmlessnesscompeting objectives
Weaponize sycophancy and rater biasagreement, authority and formatting
Inject the AI judgeposition bias and rubric spoofing

Inference-Time Bypass

Routing around a refusal at run time by stepping outside the distribution the safety training covered.

Step outside the safety distributionmismatched generalisation
Walk the threshold across turnsmulti-turn escalation
Force the opening, own the completionprefill and schema constraints
Split the token, blind the headtokeniser splits and homoglyphs
Turn over-refusal into a weaponover-refusal and alignment denial of service

Weight Surgery, Automation and Proving Grounds

Open weights and gray-box APIs, red teaming at scale, honest measurement, and where to practise with permission.

Let an attacker model run the loopPAIR, TAP, Rainbow and PyRIT
Measure with pass@k and route the class fixrates, intervals and the root cause
Practice on permitted targetsarenas, consoles and your own models

Reasoning RL (RLVR), Multimodal and Agentic Alignment

How verifier-graded reasoning, vision and audio encoders, and tool-use policies fail in ways text RLHF did not.

RLVR grades the answer, not the thoughtverifiable rewards and the ungraded scratchpad
Hack the unit test, own the verifierreward hacking in code and maths RL
Vision projectors skip text safety layerstypographic and cross-modal attacks
Audio and speech tokens bypass BPE filtersacoustic and native-audio gaps
Tool-calling RLHF trusts the schema too muchfunction-call compulsion
Multi-agent loops amplify sycophancy into consensuspeer authority and cascading agreement