Introduction
Splitting text into tokens is the actual, practical process a tokenizer performs on raw input — taking a sentence, paragraph, or document and breaking it down into the specific subword pieces a language model will process. While the previous topic covered how tokenizers are built and what algorithms they use, this topic focuses on what that splitting process actually looks like in practice, with real examples showing how different kinds of text get divided.
Seeing concrete examples of text being split into tokens makes the otherwise abstract concept of tokenization much more intuitive, and helps explain many everyday quirks developers encounter when working with token limits, costs, and prompt design.
Why Does Seeing Text Split into Tokens Matter?
Understanding the splitting process helps to:
- Build accurate intuition for how much text corresponds to how many tokens
- Anticipate why certain inputs (like code or rare words) consume more tokens than expected
- Debug unexpected token counts or context window overflows
- Write more token-efficient prompts by understanding what triggers extra splits
- Better estimate costs before sending requests to an LLM API
- Understand why the same text can split differently across models
The General Splitting Process
Example 1: A Simple, Common Sentence
Text: "The cat sat on the mat."
Likely split (illustrative):
["The", " cat", " sat", " on", " the", " mat", "."]
→ 7 tokens
Common, everyday words are frequently represented as single tokens,
since they appear often enough in training data to earn their own
vocabulary entry.Example 2: An Uncommon or Made-Up Word
Text: "supercalifragilisticexpialidocious"
Likely split (illustrative):
["super", "cal", "if", "rag", "il", "istic", "ex", "pial", "id", "ocious"]
→ 10 tokens
Since this word rarely (if ever) appears in training data as a whole,
the tokenizer falls back to combining smaller, familiar subword pieces.Example 3: Numbers
Text: "The year 2024 had 365 days."
Likely split (illustrative):
["The", " year", " ", "202", "4", " had", " ", "365", " days", "."]
→ Numbers are often split into smaller digit groupings,
which is why models can sometimes struggle with precise
arithmetic on multi-digit numbers.Example 4: Code
Text: "def calculate_total(price, tax_rate):"
Likely split (illustrative):
["def", " calculate", "_total", "(price", ",", " tax", "_rate", "):"]
→ 8 tokens
Code often tokenizes less efficiently than natural language,
since programming syntax, variable names, and symbols don't
always align neatly with the tokenizer's learned vocabulary.Example 5: Non-English Text
English: "Good morning" → often 2 tokens
Equivalent in a non-Latin script language may require
significantly more tokens for the same meaning, since many
tokenizers are trained on datasets weighted heavily toward English,
making other languages less token-efficient by comparison.What Tends to Increase Token Count
| Text Characteristic | Effect on Token Count |
|---|---|
| Common English words | Fewer tokens (often 1 per word) |
| Rare, technical, or made-up words | More tokens (split into subword pieces) |
| Numbers, especially long ones | Often split into smaller digit groupings |
| Code and special symbols | Frequently less efficient than plain text |
| Non-English / non-Latin script text | Often requires more tokens per word |
| Extra whitespace or unusual formatting | Can introduce additional tokens |
Whitespace and Punctuation Handling
Most modern tokenizers attach a leading space to the following word
as part of a single token, rather than treating spaces as fully
separate tokens:
" cat" → often one token (space + word combined)
"cat" → a different token ID than " cat"
Punctuation is frequently split into its own token:
"Hello!" → ["Hello", "!"]Practical Tools for Checking Token Splits
| Tool/Method | Purpose |
|---|---|
| Official tokenizer libraries (e.g., tiktoken) | Programmatically count and inspect exact token splits |
| Online tokenizer visualizers | Interactively see how specific text gets split |
| API response metadata | Many LLM APIs return exact token usage per request |
| Manual estimation (~4 chars/token) | Quick, rough estimate without precise tooling |
Simple Text vs Complex Text Tokenization
| Aspect | Simple, Common Text | Complex/Technical Text |
|---|---|---|
| Typical Tokens per Word | Close to 1 | Often 1.5–3+ |
| Predictability | High — matches common vocabulary | Lower — more subword splitting |
| Cost Efficiency | High | Lower |
| Examples | Everyday conversation | Code, rare terminology, non-English text |
Key Properties of the Splitting Process
- Tokenizers greedily match the longest known vocabulary pieces available for a given input.
- Common words are usually single tokens; rare or complex words split into multiple subword tokens.
- Numbers, code, and non-English text often tokenize less efficiently than everyday English prose.
- Leading spaces are frequently merged into the token that follows them, rather than counted separately.
- The exact split for any given text depends entirely on which model's specific tokenizer is used.
Where Does This Matter Most in Practice?
| Context | Practical Impact |
|---|---|
| Prompt Engineering | Understanding splits helps write more token-efficient prompts |
| Cost Estimation | Predicting token counts before sending requests to manage API costs |
| Context Window Planning | Anticipating how much actual content will fit within a token limit |
| Debugging Unexpected Behavior | Understanding why a number or rare term didn't process as expected |
| Multilingual Product Design | Planning for higher token consumption in certain languages |
Advantages of Understanding This Process
- Builds accurate, practical intuition beyond just the theoretical concept of tokenization
- Helps predict and control costs more precisely for LLM-powered applications
- Explains otherwise confusing behavior, like models struggling with certain number formats
- Supports better prompt design by avoiding unnecessarily token-heavy phrasing
- Provides a foundation for effectively working within context window constraints
Limitations
- Exact splitting behavior varies by model, making universal rules-of-thumb imperfect
- Without using an actual tokenizer tool, manual estimates remain approximate
- Splitting behavior can sometimes feel unintuitive or inconsistent to human readers
- Understanding splitting doesn't eliminate the underlying cost/context tradeoffs, only helps manage them
- Some edge cases (mixed languages, unusual formatting) can be hard to predict without direct testing
Real-World Examples
| Scenario | Splitting Behavior Insight |
|---|---|
| Sending a code snippet to an LLM API | Often consumes more tokens than an equivalent plain-English explanation |
| Translating a prompt into another language | May significantly change the total token count |
| Including long numeric IDs or data in a prompt | Can consume more tokens than expected due to digit splitting |
| Estimating cost for a customer support chatbot | Requires accounting for both common and technical vocabulary in typical queries |
| Debugging a context window overflow | Often traced back to underestimating tokens for code, numbers, or non-English text |
Best Practices
- Use an official tokenizer tool to get exact counts rather than relying solely on rough estimates.
- Expect code, numbers, and non-English text to consume more tokens per unit of content than plain English.
- Test how your specific, real-world content tokenizes before finalizing cost or context window estimates.
- Simplify or shorten prompts where possible, since unnecessary wording adds unnecessary tokens.
- Re-check token estimates when switching between different models, since tokenizers differ.
Interview Tip
A common interview question is:
"Why might a block of code or a non-English sentence use more tokens than an equivalent plain English sentence?"
A strong answer is:
Tokenizers are typically trained on large datasets, and their vocabularies end up optimized around the most frequent patterns in that data — which for most mainstream tokenizers tends to be dominated by common English words and phrases. Code and non-English text often contain symbols, syntax, or character patterns that appear less frequently in that training data, so the tokenizer can't represent them with single, efficient tokens as easily, and instead has to break them into more numerous, smaller subword pieces — resulting in a higher token count for the same amount of actual content.
Connecting it back to training data frequency ties the answer to how tokenizers are actually built.
Conclusion
Seeing concretely how text splits into tokens — from everyday sentences to code, numbers, and non-English text — turns tokenization from an abstract concept into a practical, predictable tool for managing cost, context windows, and prompt design. With tokens, tokenizers, and the splitting process now covered, the final piece of this foundation is understanding output controls, which determine how a model actually generates its token-by-token response.