Introduction

A tokenizer is the specific tool responsible for converting raw text into tokens — and back again — acting as the essential translation layer between human-readable language and the numerical format a language model actually processes. Every LLM is paired with a specific tokenizer, trained alongside or before the model itself, and that tokenizer's vocabulary and splitting rules directly shape how efficiently the model handles different kinds of text.

Understanding how tokenizers work explains many practical quirks of working with LLMs — from why certain words get split unexpectedly, to why switching between models can suddenly change token counts and costs for the exact same input text.

Why Are Tokenizers Important?

Tokenizers help to:

• Convert raw text into the numerical tokens a model can actually process
• Convert a model's numerical output back into readable text
• Define a fixed vocabulary of possible tokens the model can recognize
• Directly influence how efficiently different languages and text types are represented
• Explain differences in token counts (and cost) between different models
• Handle unfamiliar or rare words gracefully through subword splitting

The Tokenizer's Role

Whiteboard

How a Tokenizer Is Built

1. Collect a massive corpus of representative training text
2. Analyze which character sequences appear most frequently
3. Build a fixed vocabulary of common subwords, words, and symbols
   (typically tens of thousands of entries)
4. Assign each vocabulary entry a unique numeric ID
5. Define rules for splitting any new text into these vocabulary pieces

Common Tokenization Algorithms

A Simple Byte-Pair Encoding (BPE) Illustration

Starting vocabulary: individual characters
Training text contains "low", "lower", "lowest" frequently

Step 1: Notice "l" + "o" appears often → merge into "lo"
Step 2: Notice "lo" + "w" appears often → merge into "low"
Step 3: Continue merging frequent pairs...

Eventually, "low" becomes a single token, while rarer combinations
remain split into smaller pieces.

Encoding and Decoding

Encoding: Text → Tokens (numeric IDs)
"Hello world" → [15496, 995]

Decoding: Tokens (numeric IDs) → Text
[15496, 995] → "Hello world"

Every tokenizer must support both directions reliably,
since the model's raw output is just a sequence of numeric IDs.

Vocabulary Size Tradeoffs

Most modern LLM tokenizers use vocabularies in the range of roughly 30,000 to 100,000+ entries, balancing these tradeoffs.

Why Different Models Have Different Tokenizers

Each LLM is typically paired with its own tokenizer, trained on
its own representative data. This means the exact same sentence
can produce a different number of tokens depending on which
model's tokenizer is used — directly affecting cost and context
window usage differently across models.

BPE vs WordPiece vs SentencePiece

Key Properties of Tokenizers

• A tokenizer defines a fixed vocabulary and a consistent set of rules for splitting text into tokens.
• Encoding converts text to numeric token IDs; decoding converts token IDs back to text.
• Most modern tokenizers use subword algorithms like BPE, WordPiece, or SentencePiece.
• Each LLM typically uses its own specifically trained tokenizer, leading to different token counts for the same text across models.
• Tokenizer vocabulary size involves a tradeoff between sequence length and model output complexity.

Where Do Tokenizers Matter Most?

Advantages of Modern Subword Tokenizers

• Handles rare, novel, or misspelled words gracefully through subword decomposition
• Keeps vocabulary size manageable while still representing virtually any text
• Provides consistent, reversible encoding and decoding of text
• Well-established algorithms (BPE, WordPiece, SentencePiece) are efficient and battle-tested
• Enables models to generalize to text patterns not explicitly seen during training

Limitations

• Different tokenizers make direct token-count comparisons across models inconsistent
• Certain languages or specialized domains (e.g., code, scientific notation) can tokenize inefficiently
• Tokenization boundaries can sometimes split words in unintuitive or unexpected ways
• A poorly matched tokenizer can waste context window space on non-English or technical text
• Changing or retraining a tokenizer generally requires retraining the associated model as well

Real-World Examples

Best Practices

• Use the specific tokenizer associated with a model when estimating token counts or costs.
• Be aware that switching models may change token counts for identical input text.
• Consider tokenizer efficiency for your target language(s) when choosing a model for multilingual applications.
• Use official tokenizer libraries and tools rather than estimating token counts manually.
• When fine-tuning, always use the exact tokenizer paired with the base model being fine-tuned.

Interview Tip

A common interview question is:

"What is a tokenizer, and why can the same text produce different token counts across different models?"

A strong answer is:

A tokenizer is the tool that converts raw text into numeric tokens a model can process, and converts the model's numeric output back into readable text, using a fixed vocabulary built from analyzing patterns in training data. Different models are typically paired with their own tokenizer, trained on their own data using algorithms like BPE, WordPiece, or SentencePiece, which means the exact same input text can be split into a different number of tokens depending on which model's tokenizer is used — directly affecting context window usage and cost differently across models.

Naming specific tokenization algorithms makes your answer stronger.

Conclusion

Tokenizers serve as the essential bridge between human language and the numerical representations LLMs actually process, using algorithms like BPE, WordPiece, and SentencePiece to build efficient, flexible vocabularies. Understanding how tokenizers encode and decode text — and why they differ across models — completes the token-related foundation needed before exploring exactly how raw text gets split into tokens in practice