RAG document search respects who may see what when the retrieval step filters candidate documents by the requesting user's permissions before any text reaches the language model. The model should never decide access. Store source permissions with each chunk, enforce them as a query filter, and keep them synchronised with the source system.
Key takeaways
- Permissions must be enforced by the retrieval layer, not by instructions given to the language model.
- Each indexed chunk needs the access control reference of its source document stored alongside it.
- Post-retrieval filtering is a useful safety net but cannot be the only control.
- Revocations in the source system must reach the index quickly, or access lingers after it should end.
- Logging which chunks were retrieved for which user makes access reviews possible.
How does RAG document search decide what a user can see?
It decides by checking each candidate chunk against the requesting user's identity before the chunk enters the prompt. Access is a property of the source document, so the pipeline must carry that property into the index.
A typical RAG pipeline splits documents into chunks, converts them to vectors, stores them in an index, and at query time retrieves the nearest chunks to send to a language model. Each step loses context unless you design for it. A chunk stored without its source permissions is, from the index's point of view, just text, and nearest-neighbour search will return it to anyone whose question is related enough.
The authorisation decision therefore belongs to the retrieval layer. The model only sees what retrieval hands it, so whatever the prompt says about confidentiality is a request, not a control. A system prompt that tells the model to withhold salary data can be bypassed with a well-phrased question, and it leaves no audit trail. Teams that build AI assistants often start here, and the AI automation services at Rui Codex cover this kind of retrieval design as part of wider workflow work.
Where should permission checks happen in the pipeline?
Enforce them at query time, as a filter applied inside the index, and treat ingestion-time tagging as the data that makes that filter possible. Post-retrieval filtering can support the design but should not stand alone.
Option 1: Filter inside the vector query
Each chunk carries metadata such as allowed groups or an ACL reference. The query includes the user's current groups as a filter, so the index never scores chunks the user cannot read. This works well when the vector store supports metadata filtering natively. It fails when group lists become very large, or when permissions depend on relationships, such as a manager seeing a team folder, that flat metadata cannot express.
Option 2: Filter after retrieval
The system retrieves the top results, then drops any chunk the user may not read. This is easy to add to an existing pipeline. Its weakness is that the result slots are filled before filtering, so the user may receive few or no useful chunks, and the filter must also run on every path that touches the text, including logs and caches.
Option 3: Separate indexes per permission group
Some teams keep one index per department or clearance level. This gives strong isolation and simple reasoning. It fails when a document belongs to several groups, because copies drift out of sync, and it becomes costly to maintain as the number of groups grows.
Option 4: Instructions to the model
Telling the model what to withhold is not an access control. It fails once retrieval has passed the restricted text into the prompt, and it cannot be audited.
| Option | Main strength | Main failure mode |
|---|---|---|
| Query-time metadata filter | Restricted chunks never enter the candidate set | Cannot express hierarchical or relational permissions |
| Post-retrieval filter | Simple to add to an existing pipeline | Reduces useful results and leaks through unfiltered logs or caches |
| Index per permission group | Strong isolation between groups | Duplicates documents and drifts out of sync |
| Prompt instructions | No engineering effort | Not a control and cannot be audited |
How do you keep permissions current after the index is built?
Sync permission changes from the source system into the index on a defined schedule or event stream, and state the sync lag as a risk. Deleted or re-permissioned documents must be removed or updated in the index, not only in the source.
Permission changes usually originate in a document management platform, a file share or a CRM. The index needs an update process that rewrites chunk metadata, and a deletion path that removes chunks when a source is withdrawn. Identity matters as well. OAuth 2.0 access tokens carry scopes and expire, as described in RFC 6749, The OAuth 2.0 Authorization Framework, so the retrieval service should validate the caller's current token rather than trust a cached identity indefinitely. Where connectors between document stores and business systems are needed, the system integration services page describes that work.
Build the lag into the design. If the sync runs hourly, a user removed from a group keeps access for up to an hour unless the query path checks live membership. For general internal knowledge bases that trade-off is often acceptable when it is written down. For HR or legal files it usually is not, and those sources justify a live check at query time.
Does GDPR or the EU AI Act change how you should design this?
Both can shape the design, because GDPR requires appropriate security measures for personal data and the AI Act governs certain AI systems, so access control becomes part of your compliance evidence. Neither text prescribes a specific RAG architecture. GDPR Article 32 asks controllers and processors to implement appropriate technical and organisational measures for the security of processing, as set out in the official text of Regulation (EU) 2016/679. Whether an assistant falls under the AI Act's obligations depends on its purpose and category, which the official EUR-Lex text of Regulation (EU) 2024/1689 defines.
This article is general information and not legal or tax advice. Rui Codex offers technical implementation support for the access layer, not legal or compliance advice. For broader background on personal data in software systems, the security and GDPR articles on the Rui Codex blog cover related engineering topics.
What should you check before launch?
Verify that the retrieval layer, not the prompt, excludes unauthorised chunks, and that revocations propagate within the window you have documented. Run these checks in a staging index with dedicated test accounts.
- Every chunk has an access reference traceable to its source document.
- Queries from a denied test account return no chunk identifiers from restricted sources.
- Deleting or moving a source removes or updates its chunks within the documented window.
- Caches, embedding logs and conversation histories apply the same rules.
- Retrieved chunk identifiers are logged per user and per query for review.
- A named owner approves each source before it is indexed.
Hypothetical examples
Here are two hypothetical scenarios:
Scenario one. An internal assistant indexes a shared drive in which finance folders are restricted to a limited group. A sales employee asks about past pricing decisions. With a query-time filter, the finance chunks never reach the candidate set, and the answer relies only on sales documents. Without the filter, the model receives finance text and may paraphrase it, so the leak happens even though the interface looks correct.
Scenario two. A manager moves to a new team, and the source system updates group membership overnight. Until the next sync, the manager's assistant can still answer from the former team's folders. The team accepts that window for general documents but requires a live membership check for HR files, which removes the problem for the sensitive set.
Conclusion
Document search with RAG respects access rules only when permissions travel with every chunk and are enforced before text reaches the model. Filtering inside the query, keeping revocations on a documented timetable and logging retrieved identifiers together give you a design you can test and explain. Prompt instructions and post-retrieval filters can add to that design but cannot replace it.
Rui Codex can help teams plan and implement this retrieval layer as part of wider AI and integration work. Request a project consultation to discuss your sources and access model, or email info@ruicodex.com.
Frequently Asked Questions
Can a system prompt alone keep RAG answers within user permissions?
No, a system prompt cannot serve as an access control, because restricted text has already reached the model when instructions are the only barrier. Users can phrase questions to extract that text, and an instruction leaves no audit record. Enforce permissions in the retrieval query, and use prompts only as a secondary behavioural measure. Test with denied accounts and check retrieved chunk identifiers rather than generated wording.
Should each user group get its own separate index?
Usually not, because separate indexes multiply copies and create drift when documents belong to several groups. Per-group indexes suit only sources where isolation is itself a requirement. Most teams instead store access metadata with each chunk and filter at query time, which keeps one copy of each document and makes revocation a single update path.
How quickly should revoked access take effect?
The acceptable window depends on the sensitivity of each source and should be decided and documented per source. For general internal content, an hourly sync may be acceptable if the lag is stated clearly. For sensitive files such as HR or legal records, check live group membership at query time, so revocation applies to the next question rather than the next synchronisation cycle.
Does post-retrieval filtering make query-time filtering unnecessary?
No, post-retrieval filtering on its own is a weaker design. It runs after the top results are chosen, so users may receive fewer useful chunks, and any path that stores or logs unfiltered results can expose restricted text. Use it as a second check on the same permission data, while the primary filter runs inside the query.
Does the EU AI Act apply to every internal document assistant?
Not necessarily, because the Act's obligations depend on the system's purpose and category, and this article does not determine that. Check the official EUR-Lex text of Regulation (EU) 2024/1689 and take qualified legal advice for your case. Rui Codex provides technical implementation support for the access layer only.
Sources
- Regulation (EU) 2016/679 (General Data Protection Regulation) - EUR-Lex, Publications Office of the European Union
- Regulation (EU) 2024/1689 (Artificial Intelligence Act) - EUR-Lex, Publications Office of the European Union
- RFC 6749: The OAuth 2.0 Authorization Framework - Internet Engineering Task Force