Finding the right shelf in a library of a billion books
Priyanka builds KnowHub, an assistant that answers employees' questions from their own company's documents: HR policies, IT runbooks, product manuals. Four hundred companies use it. (Priyanka, KnowHub and every number here are made up for teaching.)
Picture a librarian who can glance at a shelf and say, "These three books are closest to what you described." That is vector search done by intuition. With a billion books, walking every shelf is impossible. You need a map: group books by topic, walk only the nearest few groups, and accept that once in a while you'll miss a book on a neighbouring shelf.
That is approximate nearest neighbour (ANN) search, the engine under RAG. The RAG system design guide explains why retrieval-augmented generation exists, how to chunk documents, and why hybrid search helps. This guide goes underneath and beyond. KnowHub's growing pains show each piece:
| Growing pain |
Idea it forces |
| "Search takes 3 seconds at 20 million chunks" |
Embeddings, exact vs approximate search |
| "HNSW needs 600 GB of RAM" |
HNSW, IVF, product quantisation |
| "Searching for 'ERR-4031' finds nothing useful" |
Hybrid search and reranking |
| "Adding a tenant filter returned zero results" |
Metadata filtering strategies |
| "Company A's employee saw Company B's salary bands" |
Multi-tenant, permission-aware retrieval |
| "The bot quoted last year's leave policy" |
Ingestion and freshness |
| "Is it actually any good?" |
Evaluating RAG |
1. "3 seconds at 20 million chunks": embeddings and exact search
An embedding model turns text (or images) into a vector of a few hundred to a few thousand numbers, so that similar meanings give nearby vectors.
- Comparing vectors. Use cosine similarity (the angle between vectors), the dot product or Euclidean distance. For normalised vectors, cosine and dot product rank results identically.
- Use the metric the embedding model was trained for.
- Never mix vectors from different models or versions in one index. Re-embed everything when the model changes.
Exact (flat) search compares the query with every vector. That gives perfect recall, but the cost is proportional to the number of vectors times the number of dimensions. It's fine up to perhaps hundreds of thousands, or a few million, vectors. KnowHub's 20 million chunks took 3 seconds per query, and beyond that you want ANN.
2. "600 GB of RAM": ANN indexes and their trade-offs
Every ANN index trades recall against latency, memory, and build and update cost. Recall@k means: of the true top k neighbours, what fraction did the index return?
You've reached the free preview of this post.
Subscribe to the Blogs plan (₹99/mo) — or use any Algomi Suite, Career, Investor, or Trader plan — to keep reading.
See plans
