Image (Modality)

Images are the modality that made multimodal AI mainstream — and the one that costs you the most. An image carries enormous raw data but low semantic density: a million pixels might say no more than a sentence. Working with images well is mostly about one trade-off you'll meet again and again: resolution versus tokens.

💡 In one line: Images become patches, patches become tokens — so more resolution means more detail, more tokens, and more cost.

What Makes Images Hard

• Continuous — no natural "words" to split on.
• Huge and redundant — most pixels carry little information.
• Low semantic density — 1M pixels ≈ one sentence of meaning.
• Implicit — nothing is labelled; everything must be inferred.
• 2D structure matters — position and layout mean something.

From Pixels to Tokens

The pipeline is short: image → patches → feature vectors → tokens.

A typical vision encoder splits the image into a grid of fixed-size patches (commonly 14×14 pixels). Each patch becomes one visual token.

That gives you the core equation:

tokens ≈ (width ÷ patch) × (height ÷ patch)

A 224×224 image at 14px patches → 16×16 = 256 tokens. Double the resolution to 448×448 and you get 1,024 tokens — 4× the cost for 2× the linear detail. Tokens grow with the square of resolution.

The Resolution Trade-off

This is the decision you'll make constantly:

The rule: use the lowest resolution that still contains the detail you need. Sending a 4K screenshot to ask "is this a cat?" burns thousands of tokens for a question a thumbnail could answer.

Tiling (How High-Res Actually Works)

Encoders have a fixed native resolution. To handle a big image, models tile it: split into several native-sized crops, encode each, and usually add a downscaled thumbnail for global context.

The cost is exactly what you'd expect — each tile is a full set of tokens. A document scan split into 6 tiles costs ~6× a single image. Tiling is why one page of a PDF can cost more than a page of text by an order of magnitude.

Choosing Resolution

Whiteboard

Cropping beats upscaling. If you know where the answer is, send that region — you get the detail at a fraction of the tokens.

What Models Do Well

• Scene understanding — what's in the picture, what's happening.
• OCR — reading text in images (now strong).
• Charts and diagrams — extracting trends and values.
• Document layout — tables, forms, structure.
• Captioning and visual Q&A.

What Models Still Struggle With

• Counting — especially past a handful of objects.
• Precise spatial relations — exact positions, left/right at fine granularity.
• Small text at low resolution (a resolution problem, not a reasoning one).
• Fine-grained differences — near-identical objects.
• Negation — "is there no dog?" is harder than "is there a dog?"

Visual Hallucination

The characteristic image failure: confidently describing objects that aren't there. It happens because the model has strong priors — a kitchen photo usually has a sink, so it may report one.

Mitigations: ask targeted questions rather than "describe everything," explicitly permit "not visible in the image," raise resolution when detail matters, and verify extracted values against the source.

Practical Notes

• Preserve aspect ratio — squashing distorts text and shapes.
• Format barely matters (PNG/JPEG); pixel dimensions are what drive cost.
• Multiple images multiply tokens — batch deliberately.
• Crop, don't upscale.
• Treat OCR output as a draft, not ground truth.

Summary

• Images are continuous, huge, and low in semantic density.
• Patches become tokens: tokens ≈ (W÷patch) × (H÷patch) — cost grows with the square of resolution.
• Tiling enables high-res, at ~1× tokens per tile.
• Use the lowest resolution carrying the detail you need — and crop rather than upscale.
• Watch counting, spatial precision, and visual hallucination.