Data Formatting (for Tuning)

You've collected great examples — now the model has to actually read them. Data formatting is the bridge: structuring each example into the exact template the model expects, then tokenizing it and marking which tokens to learn from. It's unglamorous, and it's where a surprising number of fine-tuning runs silently fail — the loss goes down, the model comes out wrong.

💡 In one line: Data formatting structures examples into the model's chat template, tokenizes them, and masks the loss so the model learns only the response.

Why Format Matters So Much

A model was pretrained with specific special tokens marking where turns begin and end. If your training data uses a different structure than inference will use, the model learns patterns it never sees in production — a train/serve mismatch. The symptoms are nasty: training looks fine, output is subtly broken.

Rule: format training data exactly as it will appear at inference.

JSONL: the Standard Container

Fine-tuning datasets are almost always JSONL — one JSON object per line:

jsonl
{"messages": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}
{"messages": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}

Two Common Shapes

• Chat / messages format — a list of {role, content} turns (system / user / assistant). The modern default, and required for multi-turn.
• Prompt-completion format — a flat {"prompt": ..., "completion": ...} pair. Simpler, used by some APIs and older recipes.

Chat Templates (the Critical Part)

Each model family has its own template with its own special tokens — Llama, Mistral, Qwen, and ChatML all differ. Getting them byte-exact matters, so don't hand-write them:

apply_chat_template() uses the template shipped with the model, which is the whole point — it's guaranteed to match what the model expects.

Loss Masking (the Silent Killer)

By default, training computes loss over every token — including the user's question. That teaches the model to generate questions, which you don't want.

Fix: mask the prompt tokens with -100 (PyTorch's ignore index) so only the assistant response contributes to the loss.

Most libraries (TRL's SFTTrainer with a completion-only collator) handle this — but verify it. Forgetting to mask is one of the most common quiet fine-tuning bugs.

The Formatting Pipeline

Whiteboard

Special Tokens

Watch the details that break things:

• EOS — the response must end with an end-of-sequence token, or the model never learns to stop (it rambles at inference).
• BOS — don't add it twice; apply_chat_template may already include it.
• Padding — set a pad token, and confirm padding is excluded from loss.

Length & Truncation

Every example must fit max_seq_length. Options when it doesn't:

• Truncate — risky: it can cut off the response you're trying to teach.
• Drop — cleaner for a small number of outliers.
• Split long conversations into separate examples.

Check your length distribution first — set max_seq_length to cover ~95–99% of examples rather than the single longest one, which wastes memory on padding.

Packing

Packing concatenates several short examples into one sequence to avoid wasting compute on padding. It speeds up training — but needs correct attention masking so examples don't attend across boundaries. Use your library's built-in packing rather than rolling your own.

Validate Before You Train

Cheap checks that save expensive runs:

• Decode a few formatted examples and read them — do they look exactly like inference input?
• Print the label mask — is the prompt really -100?
• Confirm EOS is present at the end.
• Check the token-length histogram.
• Verify the JSONL parses and every record has the required roles.

One hour of inspection beats a wasted GPU day.

Best Practices

• Always use apply_chat_template() — never hand-roll the format.
• Mask the prompt; train on responses only.
• Ensure EOS; set max_seq_length from the actual distribution.
• Decode and eyeball samples before launching.
• Use the same template at training and inference.

Summary

• Formatting turns clean examples into the exact structure the model expects.
• JSONL + the chat/messages format is the modern default.
• Use apply_chat_template() — each model family's special tokens differ.
• Mask prompt tokens (-100) so the model learns the response, not the question.
• Mind EOS, truncation, and packing — and decode a few samples before training.