Audio (Modality)

Audio is the modality with a hidden fork in the road. You can transcribe it to text and hand the words to an LLM — cheap, simple, and it throws away everything except the words. Or you can feed the audio itself into a model that hears it — keeping tone, emotion, speaker, and sound. Most audio mistakes come from picking the first path for a job that needed the second.

💡 In one line: Audio can be transcribed to text (losing tone, emotion, and sound) or fed natively to a model that hears it — and the choice defines what's possible.

What Audio Actually Is

A continuous waveform: air pressure over time, sampled thousands of times per second. 16,000 samples per second (16 kHz) is the standard for speech.

That's the first surprise — one second of speech is 16,000 numbers. Raw audio is absurdly redundant; almost all of that is not meaning.

What's In Audio Besides Words

This is the part transcription silently deletes:

• Prosody — pitch, rhythm, stress. "Sure." vs "Sure?" vs "Sure."
• Emotion — frustration, hesitation, enthusiasm.
• Speaker identity — who's talking, how many people.
• Overlap and turn-taking — interruptions, crosstalk.
• Non-speech sound — a door slam, a cough, a machine fault, silence.
• Environment — a call centre, a car, a cathedral.

A transcript is to audio what a caption is to an image: useful, and a massive compression of what was there.

How Audio Becomes Tokens

Raw samples are far too many, so audio is compressed before the model sees it:

1. Waveform → spectrogram (usually a mel spectrogram) — a picture of frequency over time.
2. The spectrogram is chunked into frames and passed to an audio encoder.
3. The encoder produces feature vectors, projected into the LLM's space — the same projector pattern as images.

So audio arrives as an image-like representation. A common rule of thumb: roughly 25–50 audio tokens per second, so a 10-minute recording is on the order of tens of thousands of tokens. Audio is cheap per second, expensive per hour.

The Two Paths

ASR errors compound is the underrated one: if the transcript says "recognise speech" when the caller said "wreck a nice beach," the LLM reasons confidently over the wrong words — and has no way to know.

Choosing a Path

Audio Tasks

• ASR / transcription — the workhorse.
• Diarization — who spoke when.
• Translation — speech in, another language out.
• Emotion / intent — how it was said.
• Sound event detection — alarms, faults, glass breaking.
• Voice interfaces — speech in, speech out.

Speech Out (the Other Direction)

Don't conflate audio in with audio out. TTS (text-to-speech) is a separate generative model, and speech-to-speech systems chain or fuse both directions. The latency budget for a natural voice conversation is brutal — roughly under ~300 ms to feel responsive — which is why voice systems stream rather than wait for a full response.

Challenges

• Long-form cost — hours of audio are enormous token counts.
• Noise and accents — ASR quality varies sharply.
• Overlapping speakers — still hard.
• Streaming vs. batch — real-time changes the whole architecture.
• Privacy — voice is biometric data.

Practical Notes

• 16 kHz mono is plenty for speech — higher rates mostly add cost.
• Chunk long audio with overlap so you don't cut words at boundaries.
• Feed the transcript and the audio when you need both accuracy and tone.
• Keep timestamps — they're your citations back to the source.

Summary

• Audio is a continuous waveform — ~16,000 samples/second, hugely redundant.
• Beyond words it carries prosody, emotion, speaker, and non-speech sound.
• It becomes tokens via spectrogram → encoder → projector — ~25–50 tokens/second.
• Transcribe for cheap word-level work; go native when tone, speaker, or sound matters.
• Watch long-form cost, ASR error compounding, and streaming latency.