Multimodal Large Language Models (MLLMs)

A multimodal LLM is what you get when you give a language model eyes. The clever part is what the field didn't do: nobody rebuilt LLMs from scratch to handle images. Instead, they took a pretrained vision encoder, a pretrained LLM, and bolted them together with a small connector — training mostly just that bridge. That shortcut is why MLLMs arrived so fast, and it's the key to understanding how they work.

💡 In one line: An MLLM connects a pretrained vision encoder to a pretrained LLM through a projector, turning images into tokens the language model can read.

What is an MLLM?

An MLLM is an LLM extended to accept non-text input — usually images — while keeping its language reasoning intact. The LLM remains the brain; other modalities are translated into its language: tokens.

LLM = text in, text out. MLLM = image (+ text) in, text out.

The Three-Part Architecture

Nearly every MLLM has the same three pieces:

  • Modality encoder — a pretrained vision encoder (typically a CLIP/SigLIP-style ViT) that turns an image into feature vectors. Usually frozen.
  • Connector / projector — a small trainable bridge that maps those visual features into the LLM's embedding space. This is the piece that actually gets trained.
  • LLM backbone — a pretrained language model (Llama, Qwen, Mistral...) that receives visual tokens alongside text tokens and reasons over both.

The connector is the whole trick. It's usually a tiny fraction of the parameters — but it's what lets two independently-trained models talk.

How an Image Becomes Tokens

  1. The image is split into patches (like 14×14 pixel squares).
  2. The vision encoder turns patches into feature vectors.
  3. The projector maps them into the LLM's embedding dimension.
  4. They're inserted into the token sequence next to the text tokens.
  5. The LLM attends across all of them — text and image alike.

To the transformer, an image is just more tokens. That's the entire insight.

This is also why images are expensive: one image can consume hundreds to thousands of tokens, and higher resolution means more patches — so more tokens, more cost, more latency.

Connector Designs

TypeHow it worksTrade-off
Linear / MLP projectionA simple layer maps features → LLM space (LLaVA)Simple, keeps all detail, many tokens
Q-FormerA query transformer compresses to a fixed few tokens (BLIP-2)Fewer tokens, some detail lost
Cross-attentionThe LLM attends to visual features via added layers (Flamingo)Powerful, changes the LLM

Modern open MLLMs mostly favour simple MLP projectors — they work surprisingly well.

The Training Recipe

The reason MLLMs are cheap to build: you don't train the whole thing. 

Whiteboard
Whiteboard diagram
  • Stage 1 — alignment. Freeze both big models; train only the connector on image-caption pairs. Teaches the bridge to speak the LLM's language.
  • Stage 2 — visual instruction tuning. Unfreeze the LLM (often via LoRA) and train on visual instruction data (image + question → answer). Teaches it to follow instructions about images.

LLaVA showed this works with a remarkably small budget — the heavy lifting was already done by the two pretrained halves.

Why This Design Won

  • Reuses two expensive pretrained models.
  • Trains a small bridge instead of a giant model.
  • Preserves the LLM's language ability.
  • Modular — swap in a better LLM or encoder later.

Capabilities

VQA, captioning, OCR and document understanding, chart/diagram reading, visual reasoning, grounding (pointing at regions), and GUI/screenshot understanding for computer-use agents.

Limitations

  • Token cost — images are expensive; video far more so.
  • Resolution ceiling — fine print and small details get lost; tiling helps but multiplies tokens.
  • Visual hallucination — confidently describing objects that aren't in the image.
  • Spatial reasoning — counting and precise positions remain weak.
  • The encoder bottlenecks it — the LLM can't see what the encoder didn't encode.

Best Practices

  • Send the smallest resolution that preserves the detail you need.
  • Ask targeted questions rather than "describe everything."
  • Ask it to say "not visible" to curb hallucination.
  • Verify extracted values — treat OCR output as a draft.

Summary

  • An MLLM = vision encoder + projector + LLM backbone.
  • The projector is the trainable bridge — the whole trick of the design.
  • Images become patches → features → tokens, sitting alongside text in one sequence.
  • Training is two stages: align the connector, then visual instruction tune.
  • Watch token cost, resolution limits, and visual hallucination.Â