RAG Architecture
A RAG system has two pipelines: an offline indexing pipeline that prepares your knowledge base, and an online query pipeline that retrieves and generates at request time. The vector database bridges the two. Understanding this architecture — and the choices at each stage — is the key to building RAG that's accurate, fast, and maintainable.
💡 In one line: RAG has an offline pipeline that chunks, embeds, and stores documents, and an online pipeline that retrieves relevant chunks and generates an answer.
Two Pipelines
The whole system splits cleanly into indexing (offline) and retrieval + generation (online), joined by the vector database.
Indexing Pipeline (Offline)
Done ahead of time, and re-run whenever your data changes:
- Load documents from your sources.
- Chunk them into passages (with some overlap).
- Embed each chunk with an embedding model.
- Store the vectors + metadata in the vector database.
This builds the knowledge base that queries will search.
Core Components
| Component | Role |
|---|---|
| Document loaders | Pull in source data |
| Chunker | Split documents into passages |
| Embedding model | Turn text into vectors |
| Vector database | Store and search vectors |
| Retriever (+ reranker) | Find the most relevant chunks |
| Prompt template | Combine question + context |
| LLM | Generate the answer |
| Post-processing | Citations, guardrails, formatting |
Key Design Decisions
- Chunk size + overlap — too big loses precision, too small loses context.
- Embedding model + dimension — quality vs. cost vs. speed.
- top-k + retrieval strategy — hybrid search, metadata filters, reranking.
- Prompt design + context budget — how much retrieved text to include.
- Citations + guardrails — traceability and safety.
Advanced Patterns (Ahead)
Beyond the basics: query rewriting, multi-query, HyDE (hypothetical document embeddings), reranking, agentic RAG, and GraphRAG. Each targets a specific weakness in the simple pipeline.
Evaluation
Measure both halves:
- Retrieval quality — did we fetch the right chunks? (recall, precision, hit rate)
- Generation quality — is the answer grounded in them and relevant? (faithfulness, answer relevance)
Frameworks like RAGAS help automate this.
Summary
- RAG has an offline indexing pipeline and an online query pipeline, bridged by the vector DB.
- Indexing: load → chunk → embed → store.
- Query: embed → retrieve → filter/rerank → augment → generate.
- Key choices: chunking, embedding model, top-k, retrieval strategy, and prompt design.
- Evaluate both retrieval and generation — and layer in advanced patterns as needed. EOF echo created