Precision (RAG Evaluation)

Recall asks whether you found the relevant chunks; precision asks how much of what you returned was actually relevant. In RAG, irrelevant chunks are noise — they distract the LLM, waste tokens, and can trigger wrong answers. Precision measures how clean your retrieved context is.

💡 In one line: Precision measures how many of the retrieved chunks are actually relevant — the cleanliness of your context.

What is Precision?

Precision is the fraction of retrieved items that are relevant — it answers "of what we returned, how much was useful?"

Precision=relevant retrievedall retrieved\text{Precision} = \frac{\text{relevant retrieved}}{\text{all retrieved}}

The Formula

In confusion-matrix terms:

Precision=TPTP+FP\text{Precision} = \frac{TP}{TP + FP}
  • TP (true positives) — retrieved chunks that are relevant.
  • FP (false positives) — retrieved chunks that are noise.

Precision @ k

As with recall, precision is measured over the top-k results — context precision @ k.

Why It Matters in RAG

Noisy context distracts the LLM, wastes the context budget, and can cause it to focus on the wrong thing or hallucinate. High precision means clean, focused context — which usually means better, more grounded answers.

Precision vs. Recall: the Trade-off

  • Recall — completeness (of the relevant, how many retrieved).
  • Precision — cleanliness (of the retrieved, how many relevant).

Turning the k knob moves both.

Whiteboard
Whiteboard diagram

The resolution: retrieve a wide set for recall, then rerank to restore precision.

F1: Balancing Both

When you want a single number, use F1 — the harmonic mean of precision and recall. It's high only when both are high.

How to Measure

Use ground-truth relevance, an LLM-as-judge, or a framework like RAGAS (which offers a context precision metric).

Improving Precision

  • Reranking — the biggest lever; reorder and keep the best.
  • Smaller k — return fewer, higher-quality chunks.
  • Metadata filters — drop off-topic records.
  • Better embeddings / tuned hybrid weighting.

Example

If the retriever returns 5 chunks and 3 are relevant:

Precision=35=0.60\text{Precision} = \frac{3}{5} = 0.60

Trade-offs

Chasing precision with a tiny k risks low recall (missing relevant chunks). Balance the two — the standard recipe is wide retrieval + reranking to get high recall and high precision together.

Summary

  • Precision = relevant retrieved / all retrieved — a measure of cleanliness.
  • Noisy context hurts the LLM, so precision drives answer quality.
  • It trades off with recall via k; reranking restores precision after wide retrieval.
  • Use F1 to balance both in one number.
  • Improve it with reranking, smaller k, filters, and better embeddings. EOF echo created