The Citation Is the Product

I’ll treat this as regulated decision support, not autonomous medical advice. Before architecture I need to identify the authorized corpus, author/reviewer roles, document classes, validation boundary, and cost of omission versus extra content. My invariant is: a factual sentence cannot reach approval without a stable pointer to visible source evidence and the exact artifact versions that produced it.

I’ll separate the immutable AI draft from human-edited versions so we can measure the model honestly. Humans remain final authorities; the system may abstain, flag gaps, and re-draft, but never invent a source or silently rewrite an approved response.

Actors are request coordinators, medical writers, literature-search specialists, medical/legal/regulatory reviewers, approvers, corpus curators, validation engineers, tenant administrators, and auditors. Required flow: classify the question; freeze a search protocol; ingest approved PDFs; retrieve evidence with page regions; create an outline and draft; verify every sentence; preserve pre-edit and post-edit snapshots; route stage approvals; export to the customer’s document system; and assemble a reproducibility package.

Mock goals: 99.9% authoring-plane availability, RPO under 5 minutes and RTO under 2 hours; interactive edits p95 under 300 ms; literature retrieval p95 under 3 seconds; first draft under 10 minutes; seven-year retention; tenant and therapeutic-area isolation. Out of scope: diagnosing patients, choosing treatment, replacing the source-of-record document system, and training a foundation model. We integrate with those boundaries.

That worst case is 600,000 pages/day, 6.94 pages/s average and about 83 pages/s at peak. At two CPU-seconds per page, peak OCR/layout work consumes 167 cores continuously; with 30% headroom I provision about 217 cores or an equivalent autoscaling queue. We create 100,000 factual sentences and 200,000 citation edges/day.

Raw sources are about 60 GB/day or 153 TB over seven years before replicas. Content-addressed intake avoids re-OCR and duplicate storage when requests reuse the same authorized source, so I would measure the unique-digest ratio and plan both the upper bound and observed case. If an append-only decision record averages 3 MB/response, that adds 6 GB/day or 15.3 TB. Search vectors are derived and rebuildable; raw bytes, page coordinates, snapshots, approvals, and manifests are authoritative. The architecture changes with page volume, uniqueness, and retention—not API QPS.

Every object is tenant-scoped and content-addressed. A source version has a SHA-256 digest; a page render has source digest, page number, renderer version, and render digest. An evidence span stores normalized text plus page polygon and crop digest. A sentence version cites evidence-span IDs, never “the latest PDF.”

The response is an append-only version DAG. AI_DRAFT, HUMAN_EDIT, and APPROVED are distinct snapshots; a materialized head only accelerates reads. A workflow transition uses an expected-version precondition and idempotency key. Corpus publication and response approval are separate authorities.

The control plane stores tenant policy, corpus releases, workflow state, permissions, and version manifests in a transactional database with an outbox. Source bytes and immutable snapshots go to encrypted object storage. Async workers render/OCR, detect layout, chunk evidence, and populate lexical plus vector indexes scoped to a published corpus release.

A request freezes a query protocol and corpus digest. The retrieval planner produces auditable searches; a drafting worker receives only authorized evidence and an output schema. The citation verifier performs entailment and deterministic page-span checks sentence by sentence. Unsupported text is rejected or routed to the writer as a gap. Humans edit in a collaborative UI, but each stage approval snapshots content. Export uses a receiver idempotency key; the audit service composes bytes, versions, decisions, and approvals from immutable IDs.

I split claims into atomic propositions before validation. Each proposition needs one or more source spans that jointly entail it; citation coverage and source authority are separate signals. The UI highlights exactly which phrase is supported by which page region and shows conflicts. An unsupported clause cannot borrow credibility from a supported neighbor.

Visible rendered content is authoritative. Native text helps locate candidates, but we compare its geometry with rendered pixels and OCR; hidden, off-crop, or contradictory text is quarantined. The verifier records SUPPORTED, PARTIAL, CONFLICTED, or UNSUPPORTED, model/calibrator versions, evidence IDs, and rationale codes. Only supported claims can move to approval; partial claims require edit or explicit sourced qualification.

We snapshot the untouched AI output before any human sees it, then snapshot every human stage. Otherwise edits leak into “model quality.” At the content layer I measure evidence recall, unsupported-claim rate, citation correctness, omission severity, and atomic-claim F2: F2 = 5PR / (4P + R), weighting recall four times precision because an omitted safety fact can be worse than a modest extra supported sentence.

The golden set is stratified by therapeutic area, question type, document difficulty, and known failure modes; it is double-reviewed with adjudication and versioned. Two independent rubrics cover factual/evidence quality and tone/regulatory form. An LLM judge may scale scoring only after correlation, disagreement, and calibration against the frozen human set. We blind human reviewers, keep a sentinel set out of prompt development, monitor judge-control charts, and rebaseline quarterly. Human edits are weak labels until adjudicated—they may reflect preference, not truth.

I do not delete or mutate history. Corpus release R17 marks the source retracted with effective time and reason; new authoring excludes it. A reverse citation index finds every claim and approved response pinned to the old source digest. The impact job is idempotent on (retraction_id,response_version_id) and opens risk-ranked remediation cases.

The original response remains reproducible and visibly “superseded/source retracted.” High-severity cases pause distribution and notify owners; lower severity enters review. Corrected responses create new versions and preserve who approved the change. If the index is stale, the authoritative citation table drives a slower complete scan. Retraction propagation has its own SLO and completeness reconciliation.

No. Aggregate improvement cannot average away a safety regression. The release gate includes hard catastrophic-error counts and minimum slice thresholds; oncology stays on the old model. The judge/model correlation is suspicious because both may share failure modes. I freeze the rollout, send disagreements to blinded clinicians, inspect retrieval coverage separately from generation, and test whether the judge prompt or corpus shifted.

I can canary the new model only on validated low-risk slices behind a routing manifest, shadow it elsewhere, and log both outputs without showing the candidate draft to reviewers. Recovery is an atomic route-pointer rollback; existing response versions stay pinned to their original model and judge versions.

Strong consistency applies to corpus publication, workflow transitions, approvals, authorization, and export intent. Search/index freshness is eventual and always displays the indexed corpus digest. Workers consume at least once; stage keys combine tenant, immutable input digest, stage version, and parameters, so retries reuse an output or create one result. A transactional outbox prevents state/event gaps; external export resolves timeouts with receiver lookup rather than blind retry.

Admission control and weighted tenant queues separate live authoring from backfill/evaluation. Poison PDFs have bounded resource sandboxes and DLQs. ABAC checks tenant, therapeutic area, purpose, stage, and break-glass; per-tenant envelope keys and region-pinned storage isolate data; model calls use approved zero-retention endpoints or in-VPC models. Observe queue age, stage latency/error, unsupported claims, citation coverage, reviewer override, F2 by slice, judge drift, export reconciliation, audit completeness, and access anomalies. Multi-region metadata uses warm standby; immutable objects are cross-region replicated and restore is rehearsed.

First shadow retrieval against historical requests; then give writers evidence search only; then draft in reviewer-only mode; then canary a narrow therapeutic area with dual human scoring; then permit assisted authoring while final approval remains human. Every phase has rollback, frozen baselines, and predefined quality/latency/cost gates. We validate the actual signed release artifact in a clean environment, not a developer checkout.

The central trade-off is throughput versus defensible completeness. Atomic claims, page crops, immutable snapshots, and staged approval cost storage and reviewer time, but enable safe iteration and honest evaluation. I would not promise “fully autonomous authoring”; I would promise measured assistance with explicit abstention and traceable human authority.

I’ll summarize the source of truth, safety boundary, scale path, and rollout.