Retrieve the answer. Preserve the permission.

I will split the system into source authority, immutable content lineage, asynchronous enrichment, authorization-aware retrieval, and answer composition. Source connectors and tenant IAM remain authoritative for who may read. The Knowledge Base stores auditable ACL snapshots but never treats a stale vector index as permission truth.

I need to clarify source types/edits/deletes, identity and group mapping, inheritance/link sharing, document-level versus page/section ACL, residency, OCR/media needs, freshness, citation precision, legal hold, and whether models can leave the tenant/VPC. The central invariants are: no retrieval without current authorization, no factual answer without exact authorized source anchors, document content is untrusted data, and deletion covers every derivative.

Actors are end user, project admin, source admin, connector service, parser/OCR/transcription worker, search/RAG caller, agent through MCP, privacy/security admin, and auditor. Journeys: connect source and backfill; webhook/poll revisions; upload; sanitize/parse/OCR/transcribe; chunk/embed/index; hybrid search; open citation; ask grounded question; source permission change; delete/legal hold; parser/model reindex; export audit.

NFRs: P95 keyword search under 400 ms and hybrid under 900 ms, P95 RAG first token under 2.5 s excluding long model generation, new revision searchable within ten minutes, privileged revocation effective under 30 seconds, 99.95% query availability, tenant/residency isolation, reproducible citations, and deletion attestation. Building foundation OCR/embedding models and replacing source IAM are out of scope.

1. Raw bytes and source metadata are immutable per revision; “latest” is a projection, not an audit reference.
2. Every parse, token, chunk, embedding, summary, cache, citation, and prompt context carries tenant, source revision, artifact version, data class, and lineage.
3. Authorization is checked before candidate retrieval where possible and again before returning content; stale index ACL never grants access.
4. ACL/group/connector revocation can deny immediately through an authority epoch/tombstone independent of reindex completion.
5. A citation references exact revision and source coordinates; page numbers alone are insufficient after reparse.
6. Document/tool text is untrusted data and cannot grant tools, override system policy, or request other documents.
7. Deletion or legal hold is a versioned workflow over all derivatives with attested completion/exceptions.

Five hundred thousand revisions/day × a mock 20 chunks/revision is 10 million new chunk versions/day, or 116 chunks/s average; backfills and large files require far higher burst capacity. One hundred million searches/day is 1,157 QPS average and ~9,259 QPS at an 8× working-hours peak.

Three hundred million chunks × 1,536 float16 dimensions × 2 bytes is 921.6 GB raw vectors; three physical copies/index overhead is ~2.76 TB, excluding text, postings, graph metadata, ACLs, and old versions. If semantic retrieval produces 200 candidates/query, authorization/post-filtering touches 20 billion candidates/day (~231,000/s average), so ACL representation and early partition/filtering matter. I still overfetch and post-authorize because index filters can be stale.

Document is stable source identity; DocumentRevision pins provider version/content hash and raw object. ACLSnapshot records direct principals, groups, inheritance, link policy, source revision/epoch. ParseArtifactVersion stores parser/render/OCR/transcript output by revision and artifact digests. ChunkVersion stores structural path and exact source anchor; EmbeddingVersion stores model digest and vector reference. IndexGeneration lists coverage/watermarks. Tombstone and DeletionJob fence all reads and derivatives.

Uploads/connectors are idempotent by tenant+source+external object+revision. Search accepts user/workload identity, project, query, filters, snapshot cursor, and result limit; server derives allowed scope. Citation fetch re-authorizes and returns short-lived source crop/text. Ask API stores a retrieval session manifest and returns answer claims linked to citations or an explicit abstention.

Ingress authenticates tenant/project, persists the raw object or provider reference plus revision/ACL snapshot and outbox, then acknowledges. Untrusted files enter a resource-bounded no-network sanitizer/parser. Format adapters produce a canonical structure—pages/slides/DOM/code symbols/timestamps plus positioned text and media references. Chunking preserves headings/tables/code boundaries and exact anchors. PII/classification policy runs before embedding/model routing.

Keyword and vector index writers build immutable generation shards partitioned by tenant/project/source/data class and store ACL tokens/group references. At query, identity resolves current principal/group/policy epochs. The broker runs keyword/vector retrieval with pushdown, overfetches, merges/reranks, then post-authorizes each revision before returning snippets. RAG treats snippets as quoted data, asks for claim-level citations, verifies cited IDs/spans are in the authorized context, and abstains when support is missing. Clicking a citation re-authorizes against source truth.

I use layered authorization. Connectors ingest ACL snapshots for indexing and audit, but the tenant identity/policy service owns current effective access. Group membership changes increment principal/group epochs and publish high-priority invalidations. Query capabilities include subject, tenant/project, groups or entitlement reference, policy/epochs, purpose, and expiry. Cache keys include the authorization fingerprint; revoked epochs enter a deny set.

Index pushdown narrows candidates, but every candidate is post-authorized against a fresh-enough entitlement snapshot and document tombstone before snippet/context return. Citation clicks authorize again. If group/policy freshness exceeds the tenant limit or the authority is unavailable, sensitive queries fail closed rather than serving cached content. Cached answer text is treated as derived protected data and invalidated by source/ACL lineage.

For large groups, I avoid writing every principal onto every chunk. I index compact ACL IDs and group/resource bindings, cache authorized ACL sets per principal with epochs, and use bitmaps/filtering. Highly sensitive sources can use dedicated shards or online source checks.

Citations never point only to mutable chunk number or “page 7.” They pin document revision/content hash, page/slide/sheet/media segment, canonical structural path, character/token span, bounding polygon or time range, source-text hash, parser/render version, and chunk version. Old answers retain their exact parse artifacts under retention, so audit renders the historical evidence.

Reprocessing creates a new parse/chunk/index generation. A mapping job aligns old and new anchors using source coordinates, structural IDs, and content hashes; mappings are advisory and versioned. The read pointer swaps only when generation coverage and ACL checks pass. New retrieval uses the new generation; old answers do not silently rebind. If the source revision itself changes, old citations remain valid historical evidence while UI shows a newer revision exists and may require re-answer.

Citation quality is evaluated as claim support, source correctness, span precision, completeness, and click/render success—not merely whether the model emitted bracket numbers.

Visible does not mean trusted instruction. Retrieval returns a typed data object with source, classification, citation ID, and an explicit untrusted-content boundary; it is never concatenated into the system/tool instruction channel. The RAG model has no ambient search or network capability. If it asks for another retrieval/tool, the orchestrator treats that as a proposal and performs a new policy decision bound to the original user/project/purpose.

Tool egress is allowlisted and arguments are schema/resource constrained. Data-loss prevention and sensitive-source policy may redact or deny context before model use. Instruction-shaped content is detected and logged as a risk signal, but safety does not depend on the classifier being perfect. The answer generator may quote the malicious text as subject matter while refusing its directive.

Adversarial evaluation includes visible/hidden instructions, fake citations, cross-tenant IDs, encoded URLs, indirect injection through images/metadata, tool-response injection, and attempts to cite content outside the supplied context.

Deletion first writes an authoritative tombstone/revocation epoch transactionally and emits a priority invalidation. Query and citation services consult tombstones independently of G7/G8, so content becomes unavailable immediately. G8 workers check tombstones before publishing chunks; generation swap validation proves no revoked revision remains. G7 remains physically present but logically denied until cleanup.

A lineage graph enumerates raw object, parse artifacts, thumbnails/crops, chunks, embeddings, indexes, caches, retrieval sessions/prompts, summaries, exports, evaluation copies, and backups/key generations. The deletion job crypto-erases or removes each policy-required derivative and records coverage. Legal-hold objects move to a separately encrypted, access-restricted retention class with minimal scope; ordinary search/RAG cannot retrieve them. The attestation lists completed items and explicit hold exceptions.

Cache invalidation is source-ID/epoch based, not a best-effort text search. Reindex/backfill jobs use lower priority and cannot resurrect tombstoned revisions.

Connector receipts and raw revisions commit durably with outbox; parsing/chunking/embedding/indexing are at-least-once and idempotent on (revision, stage code, artifact manifest). Per-tenant/source queues, retry budgets, circuit breakers, and quarantine isolate poison files. Live revisions outrank backfill/reindex; large files are chunked and checkpointed.

If embedding/vector retrieval is down, keyword search with the same authorization and citation path can serve. If parsers lag, show freshness/coverage and continue serving last valid non-revoked revision. If identity/ACL authority is uncertain beyond policy, sensitive results fail closed even if search is healthy. Search availability and authorization availability are separate SLOs.

Generation build is immutable and validated, then an atomic pointer swap; compaction/deletion respect tombstones. Queue messages carry opaque IDs and classifications, not entire documents.

Health: connector cursor/ACL lag, raw ingest loss/duplication, parser/OCR failure, poison/DLQ, stage queue age, index coverage/generation lag, vector/keyword latency, authorization latency/denials/freshness, cache invalidation, citation render success, deletion coverage, and cost. Quality: retrieval recall@K and precision/NDCG by format/language/source, ACL false allow/deny, claim citation precision/recall, unsupported claim rate, answer completeness, freshness, abstention, and user resolution.

Rollout starts with upload plus exact keyword search, then source citations, then hybrid retrieval, then answer drafting with mandatory citations, then selected agent/MCP access. Every phase uses synthetic cross-tenant canaries, revoked-access tests, adversarial documents, golden query sets, human claim adjudication, shadow reindex, kill switches, and rollback to previous generation. No automation is promoted on aggregate answer helpfulness alone.

Atlas Vault stores immutable document revisions, ACL snapshots, parse artifacts, chunks, embeddings, and citation anchors with complete lineage. Connectors and file parsers are asynchronous and idempotent; immutable index generations swap only after validation. Query resolves current identity/epochs, pushes ACL filters into keyword/vector retrieval, overfetches, and post-authorizes every revision before snippet or model context.

RAG sees typed untrusted evidence with no ambient tools and must produce claim-level authorized citations or abstain. Tombstones deny immediately while deletion traverses every derivative and respects isolated legal hold. The trade-off is recall/latency versus authorization certainty: I spend on overfetch and post-filtering, and fail closed when ACL freshness is uncertain.