RAG System Design: How to Ground an AI's Answers in Your Own Data

1. A confident answer is not the same as a known answer

Imagine the same school library assistant from our evals and observability guide. A member asks, "Can I renew a book that's on hold for someone else?" The library changed this policy last month. The assistant, trained on general language patterns, has never seen this specific library's rules. It still answers fluently: "Yes, you can renew it as many times as you like."

The answer is wrong, and confidently so. A large language model, or LLM, generates the words that are statistically likely to follow, not the words that are verified true. When a fact lives outside its training data — a private policy, a changing price, this week's inventory — the model either says it does not know or, more dangerously, invents something plausible. That invention is commonly called a hallucination.

Retrieval-augmented generation, or RAG, is an architecture that fetches relevant information from a trusted source at the moment of the question and gives it to the model as part of its input, so the model answers from evidence instead of memory.

Our running example continues to be the library assistant, now answering questions from the library's own policy documents, catalog records, and notices. All figures below are invented teaching examples unless a source is linked.

2. The idea in simple words

Think of an open-book exam versus a closed-book one. A closed-book exam tests what you memorized. An open-book exam tests whether you can find the right page and use it correctly. RAG turns the model's exam from closed-book to open-book.

Two things happen at question time:

Retrieve. Search a document collection for the passages most relevant to the question.
Generate. Give the model the question plus those passages, and ask it to answer using only that evidence.

This requires the documents to be prepared in advance so they can be searched quickly. That preparation step is called indexing, and it happens before any question is ever asked.

Term	Meaning
Corpus	The full collection of documents available to retrieve from
Chunk	A smaller passage cut from a document, small enough to retrieve and read individually
Embedding	A list of numbers that represents a chunk's meaning, positioned so similar meanings sit close together
Vector index	A data structure that finds the chunks whose embeddings are closest to a query's embedding, without comparing against every chunk one by one
Retriever	The component that turns a question into a ranked list of chunks
Grounding	Basing an answer on retrieved evidence rather than on the model's memorized knowledge

3. How it works: two pipelines, not one

RAG has an offline pipeline that runs whenever documents change, and an online pipeline that runs on every question.

flowchart TD
    subgraph Offline indexing
    D@{ icon: "algomi:ext-doc", label: "Library documents", pos: "b", h: 52 }
    C@{ icon: "algomi:async-gear", label: "Split into chunks", pos: "b", h: 52 }
    D --> C
    EM@{ icon: "algomi:svc-spark", label: "Create embeddings", pos: "b", h: 52 }
    C --> EM
    IX@{ icon: "algomi:data-cylinder", label: "Vector index", pos: "b", h: 52 }
    EM --> IX
    end
    subgraph Online query
    Q@{ icon: "algomi:ext-person", label: "Member question", pos: "b", h: 52 }
    QE@{ icon: "algomi:svc-spark", label: "Embed the question", pos: "b", h: 52 }
    Q --> QE
    R@{ icon: "algomi:svc-search", label: "Search the index", pos: "b", h: 52 }
    QE --> R
    IX -.-> R
    K@{ icon: "algomi:svc-cube", label: "Top chunks", pos: "b", h: 52 }
    R --> K
    P@{ icon: "algomi:svc-spark", label: "Build prompt: question + chunks", pos: "b", h: 52 }
    K --> P
    M@{ icon: "algomi:svc-spark", label: "Model generates an answer", pos: "b", h: 52 }
    P --> M
    end

Reading the diagram. The two boxes never run at the same time.

Offline indexing (top) is the librarian preparing the reading room before opening: documents are split, each piece is turned into an embedding, and the result is stored in the vector index.
Online query (bottom) happens while the member waits. The question is embedded the same way, the index is searched, and the top chunks go into the prompt beside the question.

The only link between the two is the dashed arrow from the index to the search step. The online side reads what the offline side prepared, so if the offline side is late, the online side answers from old pages.

The offline pipeline never runs during a live request; it keeps the index current as documents are added, changed, or removed. The online pipeline runs in the seconds a member is waiting for an answer, so every step in it is on the latency budget.

The offline pipeline as movement. Everything here happens before a member asks anything:

Offline: documents are split, embedded, and stored before any question arrives · hover or tap to pause

You've reached the free preview of this post.

Subscribe to the Blogs plan (₹99/mo) — or use any Algomi Suite, Career, Investor, or Trader plan — to keep reading.

See plans

RAG System Design: How to Ground an AI's Answers in Your Own Data

1. A confident answer is not the same as a known answer

2. The idea in simple words

3. How it works: two pipelines, not one

Sources