Audio and speech tokens bypass BPE filters

Cascaded Voice Pipelines vs. Native "Omni" Audio Models

There are two completely different ways AI systems process voice and .wav / .mp3 audio files, and each has its own unique RLHF failure mode¹:

1. Architecture A (Cascaded ASR to LLM): A speech-to-text model (like Whisper) transcribes your audio into a text string, and passes that text string to a standard text LLM.
2. Architecture B (Native Audio / Omni Models, e.g., Gemini Live/Flash Audio, GPT-4o Realtime): The audio waveform is never converted into text BPE tokens at input! Instead, a neural audio codec (or continuous mel-spectrogram encoder) slices the waveform into 20ms–40ms acoustic frames and feeds them directly into the Transformer's residual stream alongside text tokens.

Why Native Audio Tokens Break Text-Trained RLHF

When a native Omni model ingests a .wav file directly into its residual stream, three alignment assumptions break down simultaneously:

1. Zero BPE Keyword Collisions

In Card 3, Lesson 05, we had to use Unicode homoglyphs or string concatenation to stop the BPE tokenizer from emitting a flagged integer token ID. In a native audio model, there are no BPE text tokens for the spoken input!

• A 3-second spoken word is represented as 75 to 150 continuous acoustic frame vectors (or Residual Vector Quantization RVQ codebook IDs) capturing pitch, timbre, formants, and background noise.
• Because 99% of the model's safety preference dataset was collected on text BPE tokens, the acoustic token sequence travels through early layers without tripping BPE-anchored safety heads.

2. Acoustic Prosody & Emotional Urgency Hijacking

Text prompts are emotionally flat, if you type "I AM IN AN EMERGENCY", the model just sees uppercase ASCII tokens. Native audio models are explicitly trained to attend to paralinguistic acoustic features: vocal tremor, rapid breathing, authoritative cadence, and emotional distress.

• During multimodal SFT/RLHF, human raters heavily penalize voice assistants that sound cold, robotic, or unhelpful when a speaker's voice sounds stressed or authoritative.
• Red teamers exploit this by synthesizing audio prompts (via TTS with prosody control) using a calm, rapid, authoritative incident-commander cadence, or high-urgency vocal stress, which spikes the model's empathy/helpfulness weights and suppresses refusal.

3. Acoustic Perturbations & Background Layering (Cocktail-Party Smuggling)

Human listeners and standard safety monitors focus on the loudest foreground voice in an audio clip. However, high-capacity multimodal Transformers can separate and attend to overlapping audio streams ("the cocktail-party effect") or high-speed time-compressed speech.

(redacted: a table of three acoustic smuggling techniques, time-compressed speech, dual-channel foreground/background mixing, and phonetic/acrostic spelling, with the construction details for each.)

Attacking Cascaded Pipelines (ASR Hallucination & Homophone Smuggling)

Even when a target uses Architecture A (Cascaded Whisper ASR to Text LLM), the Speech-to-Text transcriber introduces a translation seam you can exploit:

• Technical Homophones: Spoken audio has no spelling. If you speak a prompt where a restricted tool name or command sounds identical to a benign phonetic spelling, Whisper transcribes the phonetic variant, bypassing exact-match keyword guardrails on the text string while the downstream LLM still understands the phonetic context.
• Polyglot Audio Code-Switching: Speaking the conversational framing in one language and switching mid-sentence to English technical terms confuses single-language ASR safety wrappers while multilingual Omni models parse both seamlessly.

Generating Dual-Stream & Time-Compressed Audio Probes in Python

(redacted: a Python/ffmpeg helper that time-compresses a spoken payload and mixes it quietly under a benign foreground track.)

Field Exercise 5.04: Audio Attachment vs. Text Prompt

(redacted: an exercise that ran against third-party hosted Omni models (Google AI Studio) by uploading a spoken or time-compressed .wav payload. Replace with a version run on an open-weights audio model you operate.)

Next Lesson

We've seen how RLVR reasoning (01–02), Vision (03), and Audio (04) create brand-new seams. What happens when an aligned model is given Tools & Function Calling (tools=[...] / MCP servers)? In 05 Tool-calling RLHF trusts the schema too much, we exploit Function-Call Compulsion and parameter-level safety blindness.