Multi-Tenant RAG: Retrieving the Right Answer from the Right Customer

1. The apartment building

Picture an apartment building with a helpful concierge at the front desk. Hundreds of families live there. Each family keeps private papers in its own locked cabinet in its own apartment. Residents walk up and ask the concierge questions such as "What did my lease say about pets?"

The concierge is clever and fast, but there is one rule that matters more than speed: the concierge may only open the cabinet that belongs to the person asking.

That building is our running example for the whole article.

RAG (retrieval-augmented generation) means the assistant looks up relevant passages from stored documents and writes its answer from them. The basics are in the RAG system design guide. This article is about what changes when many customers share one system. The short version is in section 4 of the vector search guide; here we go deeper.

2. The problem: similarity is not permission

Suppose two customers each upload an employee leave policy.

• Acme Corp: 25 days of leave.
• Globex Inc: 15 days of leave.

An Acme employee asks, "How many leave days do I get?" Both documents are about the same topic, so in embedding space (the numeric map RAG searches) they sit almost on top of each other. A nearest-neighbour search that does not know about tenants may return Globex's policy.

An embedding captures meaning. It does not capture ownership. Being similar to the question does not mean the user is allowed to see it.

Where the analogy stops: a human concierge knows who lives where. A vector index only knows numbers, so ownership must be stored and enforced explicitly.

In an enterprise product this is not a small bug. It is a data breach, and the model will happily quote whatever it is handed.

3. The rule: carry the tenant everywhere

One rule drives the design:

Resolve the tenant from the authenticated session once, then carry it through every stage: ingestion, retrieval, context assembly, caching and audit.

The tenant must come from the signed login token, not from a field the client can edit. A request that says ?tenant=globex is like a resident telling the concierge which apartment to open.

flowchart TD
    A@{ icon: "algomi:ext-person", label: "User request + signed token", pos: "b", h: 52 }
    B@{ icon: "algomi:svc-cube", label: "Resolve tenant from token", pos: "b", h: 52 }
    A --> B
    C@{ icon: "algomi:svc-decision", label: "Cache lookup\nkey contains tenant", pos: "b", h: 52 }
    B --> C
    H@{ icon: "algomi:svc-spark", label: "Return answer", pos: "b", h: 52 }
    C -- hit --> H
    D@{ icon: "algomi:svc-search", label: "Vector search\nwith mandatory tenant filter", pos: "b", h: 52 }
    C -- miss --> D
    E@{ icon: "algomi:svc-cube", label: "Check every chunk\nbelongs to this tenant", pos: "b", h: 52 }
    D --> E
    F@{ icon: "algomi:svc-spark", label: "Rerank and build prompt", pos: "b", h: 52 }
    E --> F
    G@{ icon: "algomi:svc-spark", label: "Model answers", pos: "b", h: 52 }
    F --> G
    I@{ icon: "algomi:sec-key", label: "Store in cache\nunder tenant key", pos: "b", h: 52 }
    G --> I
    I --> H