Introduction

Tokens are the fundamental units of text that language models actually read, process, and generate — not individual characters, and not always whole words, but chunks of text determined by a model's tokenizer. Before any text reaches an LLM, it must first be broken down into these tokens, converted into numerical representations the model can work with mathematically.

Understanding tokens is essential for working effectively with LLMs, since nearly everything measurable about a model — context window size, API pricing, and generation speed — is counted and billed in tokens rather than words or characters.

Why Do Tokens Matter?

Tokens help to:

• Serve as the actual unit of text that language models process internally
• Determine how API usage and pricing are measured and billed
• Define how much content fits within a model's context window
• Explain why word count and token count don't always match
• Influence response generation speed, since models generate one token at a time
• Provide the foundation for understanding tokenization, covered in more depth separately

From Text to Tokens: The Basic Flow

Whiteboard

What Counts as a Token

Tokens aren't always whole words. Common patterns include:

"cat" → 1 token (a common whole word)
"unbelievable" → might split into "un", "believ", "able" (3 tokens)
"ChatGPT" → might split into "Chat", "G", "PT" (3 tokens)
" " (a space) → often merged with the following token
Punctuation → often its own separate token

A rough rule of thumb: 1 token ≈ 4 characters of English text,
or about 0.75 words per token on average.

Why Not Just Use Whole Words as Tokens?

Subword tokenization allows models to represent rare or made-up words by breaking them into smaller, familiar pieces, rather than treating every unfamiliar word as completely unknown.

Tokens and the Context Window

A model's context window limit (e.g., 128,000 tokens) is measured
in tokens, not words or characters.

This means the exact same context window can hold:
- More tokens of simple, common English text
- Fewer tokens of dense technical text, code, or non-English languages,
  which often require more tokens per word

Tokens and API Pricing

Most LLM APIs charge based on the number of input and output tokens processed, rather than a flat per-request fee.

Example pricing structure (illustrative):
Input tokens:  $X per 1,000 tokens
Output tokens: $Y per 1,000 tokens

A longer prompt or a longer generated response directly
increases the total cost of a single request.

Words vs Tokens vs Characters

How Token Count Varies by Language

Key Properties of Tokens

• Tokens are the actual unit of text that LLMs read, process, and generate — not raw characters or whole words.
• Most modern LLMs use subword tokenization, balancing vocabulary size and flexibility.
• Context window limits, API pricing, and generation speed are all measured in tokens.
• Token count doesn't map 1:1 to word count — it varies based on language, vocabulary, and text complexity.
• Models generate responses one token at a time, sequentially, during inference.

Where Do Tokens Matter Most?

Advantages of Subword Tokenization

• Handles rare, misspelled, or made-up words gracefully by breaking them into familiar pieces
• Keeps vocabulary size manageable compared to whole-word approaches
• Provides a consistent, mathematically usable representation of text for the model
• Balances efficiency with flexibility across many different languages and domains
• Enables models to generalize to words they've never seen exactly before

Limitations

• Token count doesn't map intuitively to word or character count, which can confuse cost/length estimation
• Different languages can be tokenized with very different levels of efficiency
• Long, unfamiliar, or highly technical words can consume more tokens than expected
• Token-based billing can make costs harder to predict without careful estimation
• Tokenization boundaries can sometimes split words in ways that feel unintuitive to humans

Real-World Examples

Best Practices

• Estimate token counts (using a tokenizer tool) before sending large inputs to an API, to manage cost and limits.
• Write concise, focused prompts to avoid unnecessary token consumption.
• Account for non-English text potentially requiring more tokens per word when planning applications.
• Remember that both input and output tokens typically count toward usage and cost.
• Use official tokenizer libraries (e.g., tiktoken) to get accurate token counts for a specific model.

Interview Tip

A common interview question is:

"What is a token in the context of language models, and why isn't it the same as a word?"

A strong answer is:

A token is the actual unit of text that a language model processes, determined by its tokenizer, and it's often a subword piece rather than a whole word — common words might be a single token, while longer or rarer words get split into multiple smaller pieces. This differs from word count because tokenization is designed to balance vocabulary size with flexibility, allowing the model to represent any word, including ones it's never seen before, by combining smaller, familiar subword units — which is why token count and word count don't map directly to each other.

Explaining subword tokenization's purpose (handling rare words) makes your answer stronger.

Conclusion

Tokens are the essential building blocks that language models actually read, process, and generate, underlying everything from context window limits to API pricing and response speed. Understanding how text becomes tokens — and why token count differs from word count — provides the necessary foundation for the next topic: tokenizers, the specific tools responsible for performing this text-to-token conversion.