Vector Search in RAG
Vector search is the default retriever in RAG. You embed the user's query, find the nearest chunk vectors in your knowledge base, and hand those top matches to the LLM as context. Because it matches on meaning rather than exact words, it's the natural starting point for retrieval — and understanding its strengths and gaps is what motivates every other method in this section.
💡 In one line: Vector search embeds the query and returns the most semantically similar chunks — RAG's baseline retrieval method.
What is Vector Search in RAG?
Also called semantic search or dense retrieval, vector search compares the query embedding against the chunk embeddings and returns the closest ones. It's "dense" because it uses dense vectors (embeddings), as opposed to sparse keyword counts.
How It Works
Remember: the query is embedded with the same model used for the chunks.
Why It's the Default
- Semantic — matches meaning, so "reset my password" finds "recover login credentials."
- No keyword overlap needed — robust to synonyms and paraphrase.
- Simple — one embedding model, one ANN index, one query.
Strengths
- Great at conceptual / paraphrased queries.
- Language-flexible with multilingual models.
- Fast at scale via ANN.
Weaknesses — and What Fixes Them
Vector search alone has real gaps. Each one is addressed by a method later in this section:
| Gap in plain vector search | Fixed by |
|---|---|
| Misses exact terms (codes, names, IDs) | Hybrid Search |
| Ranking isn't perfect | Reranking |
| Chunks lose surrounding context | Parent-Child Retrieval |
| A single, ambiguous query | Multi-Query Retrieval |
| No multi-hop / relational reasoning | Graph RAG |
So vector search is the foundation — the others build on it.
Key Knobs
- top-k — how many chunks to retrieve.
- Distance metric — usually cosine.
- Embedding model — quality and domain fit.
- Chunk quality — good chunking makes retrieval far better.
Code Example
When Vector Search Is Enough
For conceptual Q&A over reasonably clean documents, plain vector search often works well. Reach for the other methods when you need exact-term matching, higher precision, more context, or relational reasoning.
Summary
- Vector search is RAG's default retriever — embed the query, return the nearest chunks.
- It matches on meaning, so it's robust to synonyms and paraphrase.
- Its gaps motivate hybrid search, reranking, parent-child, multi-query, and graph RAG.
- Tune top-k, metric, embedding model, and chunk quality.
- It's the foundation every other retrieval method builds on. EOF echo created