Data Collection (for Tuning)
Fine-tuning has one input that matters more than the model, the method, or the hyperparameters: your data. Whatever you feed the model gets baked into its weights — good patterns and bad ones. Data collection is where you decide what those patterns will be, and it's the stage where most fine-tuning projects are quietly won or lost.
💡 In one line: Data collection gathers the examples your model will learn from — and their quality sets the ceiling on everything that follows.
Why This Step Dominates
Fine-tuning is pattern imitation. The model doesn't judge your examples — it copies them. So:
- Inconsistent formats → inconsistent output.
- A few wrong answers → learned mistakes.
- Sloppy tone → sloppy tone, permanently.
Garbage in, garbage baked in. You can't prompt your way out of bad training data — you have to retrain.
What an Example Looks Like
For instruction tuning, each example is an (instruction → ideal response) pair:
json
{
"messages": [
{"role": "system", "content": "You are a support agent for Acme."},
{"role": "user", "content": "My order hasn't arrived."},
{"role": "assistant", "content": "I'm sorry about that. Could you share your order number so I can track it?"}
]
}The assistant turn is the target — it should be exactly what you want the model to say.
How Much Data?
Less than people expect:
| Amount | What it's good for |
|---|---|
| 50–100 | A quick sanity check; format only |
| 500–1,000 | A real starting point — often enough |
| 1,000–10,000 | Solid task performance |
| 10,000+ | Diminishing returns, unless the domain is broad |
LIMA made the point sharply: ~1,000 carefully curated examples produced a strong instruction-follower. A few thousand excellent examples beat a hundred thousand mediocre ones.
Where the Data Comes From
- Existing logs — support tickets, chat transcripts, past outputs. The best source if you have it — it's real and already in your domain.
- Human-written — experts author ideal responses. Highest quality, slowest, most expensive.
- Synthetic — an LLM generates examples (self-instruct, as in Alpaca), or a stronger model produces responses (distillation). Fast and cheap, but inherits the teacher's flaws — and check the licence terms.
- Public datasets — FLAN, Dolly, OpenAssistant. Good for general instruction-following, rarely for your domain.
- Hybrid — the common real-world answer: synthetic draft → human review.
Choosing a Source
What "Quality" Actually Means
- Correct — the response is genuinely right.
- Consistent — same format, same tone, same conventions every time. Inconsistency is worse than imperfection.
- Diverse — covers the real spread of inputs, including edge cases.
- Representative — matches what production traffic actually looks like.
- Clean — no PII, no boilerplate, no artefacts.
Consistency is the underrated one. A model trained on three different answer styles learns to pick one at random.
Diversity Beats Volume
A thousand near-identical examples teach the model one narrow thing. A thousand varied examples — different phrasings, lengths, difficulties, edge cases — teach the underlying skill. Deduplicate aggressively; near-duplicates inflate your count without adding signal.
Common Pitfalls
- Only happy paths — no failure cases, so the model never learns to say "I don't know."
- Inconsistent formatting across examples.
- Leaked PII or confidential data, now inside the weights.
- Licensing — some model outputs can't legally train competing models.
- No held-out set — you can't tell whether it worked.
- Class imbalance — 90% of one intent, so it defaults to that.
Splits & Governance
Split before you train: train / validation / test (roughly 80/10/10). The test set must stay untouched — it's your only honest read.
And treat data as an asset: version your datasets, log provenance (where each example came from), and document collection decisions. When a tuned model misbehaves six months later, the dataset is where you'll look.
Best Practices
- Start with 500–1,000 excellent examples; scale only if evaluation says you must.
- Prefer real data over synthetic; use synthetic + human review as the fast path.
- Deduplicate, enforce one consistent format, and include edge and refusal cases.
- Strip PII and check licences.
- Hold out a test set before anything else.
Summary
- Data quality sets the ceiling on fine-tuning — the model copies what you give it.
- 500–1,000 excellent examples is a real starting point; quality beats quantity (LIMA).
- Sources: real logs > human-written > synthetic — or synthetic drafted, human reviewed.
- Consistency and diversity matter more than volume; deduplicate aggressively.
- Split first, strip PII, check licences, and version the dataset. EOF echo created