Metadata Storage

A vector database stores more than vectors. Alongside each embedding you can store metadata — attributes like source, category, date, author, or permissions. Metadata is what turns raw similarity search into precise, filtered, production-ready retrieval: you don't just find similar results, you find similar results that also match the right conditions.

💡 In one line: Metadata is the structured attributes stored with each vector — enabling you to filter similarity search by things like category, date, or user.

What is Metadata?

Metadata (sometimes called the payload) is the set of structured attributes saved with each vector: the original text, source, category, tags, timestamps, IDs, and permissions — anything you might want to filter on or display.

Why Metadata Matters

  • Filtering — restrict search to matching records (category, date, user).
  • Returning content — the vector isn't readable; the metadata holds the text to show or use.
  • Access control & multi-tenancy — filter by user, tenant, or permissions.
  • Faceting, freshness, and deduplication.

Metadata Filtering (the Key Feature)

The standout capability: combine similarity search with metadata conditions. For example — "find similar chunks and category = cooking and date after 2024." Only records that pass both the similarity and the filter are returned.

The Filtered-Search Flow

Whiteboard
Whiteboard diagram


Pre-filter vs. Post-filter

There are two ways to combine a filter with search:

  • Pre-filter — apply the metadata filter first, then search within that subset. Accurate, but can be slower.
  • Post-filter — search first, then drop non-matching results. Fast, but may return fewer than the requested top-k.

Different databases handle this differently — it's worth knowing which yours uses.

What to Store as Metadata

Common fields: source, author, date/timestamp, category/tags, language, permissions/tenant, chunk position, and the original text. Keep it lean — store only what you'll filter on or return, to avoid bloating the index.

Code Example


 
Best Practices
 
     
  • Index the metadata fields you filter on.
    •  
  • Keep payloads lean — don't store what you won't use.
    •  
  • Always store the original text for RAG.
    •  
  • Use metadata for access control and multi-tenancy.
    •  
  • Know whether your DB does pre- or post-filtering.
    •  
 
Summary
 
     
  • Metadata is the structured attributes stored with each vector (the payload).
    •  
  • Its killer feature is filtering — combining similarity with conditions like category or date.
    •  
  • It also holds the original text to return, and enables access control and multi-tenancy.
    •  
  • Pre-filter is accurate; post-filter is fast but can return fewer results.
    •  
  • Store only what you'll filter or return, and index your filter fields. EOF echo created