Text (Modality)

Text is the first modality — and in multimodal AI, it's the privileged one. Every other modality gets translated into text's world: images become tokens, audio becomes tokens, and the answer comes back as words. Text isn't just one of the modalities in an MLLM; it's the anchor the whole system is organised around.

💡 In one line: Text is the anchor modality of multimodal AI — discrete, compositional, and the space every other modality is projected into.

Why Text Is Different

Other modalities are continuous signals — pixel intensities, sound pressure over time. Text is discrete symbols drawn from a finite vocabulary. That difference shapes everything:

Text is already "compressed meaning." A photo needs a million pixels to say what "a red car" says in three tokens. That density is why text is so cheap relative to images — and why it makes such a good interface.

Text as the Pivot

In an MLLM, the LLM backbone's embedding space is a text space — it was pretrained on text. So the projector's job is literally: translate visual features into something that looks like text embeddings.

Every other modality bends toward text. That's not an accident — it's because:

• Pretrained LLMs already carry vast world knowledge, in text.
• Text is the instruction channel — how you say what you want.
• Text is the output for nearly all MLLMs.
• Text is human-readable — the natural interface both ways.

Text's Four Roles

Text does more jobs than any other modality.

Whiteboard

The fourth is the subtle one: paired captions are what teach the model that a photo of a dog and the word "dog" belong together. Text supervises the other modalities.

How Text Becomes Tokens

Tokenization splits text into subword units from a fixed vocabulary, each mapped to an embedding. Rough rule: ~4 characters ≈ 1 token in English (more for other scripts and for code). (See the tokenization subtopics for detail.)

The key contrast: text tokenization is lossless and reversible — you can decode tokens back into the exact original string. Image "tokenization" is lossy — patches → features → you can't reconstruct the photo.

Text Tasks in a Multimodal World

• The instruction — "what's in this chart?"
• OCR output — text extracted from images (text as a target).
• Captions — text describing other modalities.
• Documents — text and images interleaved, as they naturally occur.
• Code — a specialised text sub-modality with its own conventions.

Strengths

• Cheap — few tokens per unit of meaning.
• Precise — exact words, exact instructions.
• Composable — infinite meanings from finite symbols.
• Auditable — you can read what the model read.

Limitations

• Can't convey everything — spatial layout, tone of voice, a face.
• Ambiguity — words are underspecified without context.
• Lossy about the world — a caption throws away most of the image.
• Language coverage — tokenizers and training data are English-heavy, so other languages cost more tokens for the same meaning.

Why It Still Anchors Everything

Even as models grow more capable across modalities, text stays central because it's the modality that is simultaneously compressed, explicit, human-readable, and backed by the largest pretrained models. Everything else is easier to translate into text than the reverse.

Summary

• Text is discrete symbols; other modalities are continuous signals.
• It's compressed meaning — very high semantic density per token.
• In MLLMs, text is the pivot: the LLM's embedding space is a text space.
• It plays four roles: input, instruction, output, and the glue that aligns other modalities.
• Its limits are real (layout, tone, non-English cost) — but it remains the anchor.