Receipts Without Forever

I’ll define immutability as: no actor can silently rewrite the history that remains under policy. It does not mean retain every sensitive byte forever. I’ll separate a minimal append-only receipt ledger from encrypted payload objects, derived indexes, and retention metadata. Deletion appends a verifiable tombstone and completion attestation, then removes or crypto-erases in-scope payloads according to lineage; legal hold narrows and overrides that workflow with explicit authority.

We record observable inputs, outputs, versions, permissions, tool effects, approvals, and reason codes—not hidden model chain-of-thought, which may be unavailable, sensitive, or misleading.

Actors: application/agent services, human approvers, tenant admins, security responders, privacy/deletion officers, legal-hold custodians, auditors, and platform SREs. Required: durable receipt ingestion; event correlation; context/tool/permission manifests; payload redaction/classification; hash-chain and checkpoint verification; search and timeline views; scoped exports; retention schedules; subject/artifact lineage; deletion and hold workflows; backup purge/expiry; privileged-access audit; and completeness reconciliation against producers.

Mock NFRs: 50M actions/day, 12 receipt events/action, 99.99% ingest availability, durable ack p99 under 200 ms, searchable within 60 seconds, seven-year maximum policy with tenant overrides, standard deletion completion under 30 days, export of one action under two minutes, zero cross-tenant access. Out of scope: storing private reasoning, proving that a model’s claim is true, guaranteed unlearning from third-party foundation models, and using the audit plane as the operational workflow database.

That is 600M events/day, 6,944 events/s average; at a 10× burst, roughly 69K/s. Logical ingress is 1.2 TB/day. At 4:1 compression and three replicas, physical ledger storage is about 900 GB/day, 328.5 TB/year, or 2.3 PB over seven years—before indexes and payload blobs. Indexing every field would be financially and privacy-prohibitive.

So the hot search tier keeps a bounded window and selected low-cardinality metadata; older partitions remain columnar/WORM and are queried asynchronously. Large prompts, context, tool results, and outputs live as encrypted content-addressed objects with retention classes; ledger events carry hashes and minimal safe metadata. Sampling is unacceptable for compliance events, but detailed telemetry can be sampled separately.

An Action gets a tenant-scoped sortable ID and parent/root correlation IDs. Each producer owns an independently ordered stream identified by (tenant_id, producer_id, stream_epoch); the primary append key adds sequence_no. A reset must open a new signed epoch rather than reuse sequence numbers. Each ReceiptEvent also includes event type, actor/service/workload identity, timestamp and trusted ingest time, policy decision ID, permission snapshot digest, input/context/output object digests, model/provider/endpoint/config, prompt-template digest, tool name/version/arguments digest/result digest, approval IDs, side-effect business key/receipt, previous-event hash, and canonical event hash.

Payload objects have classification, subjects/artifacts, region, envelope-key reference, retention schedule, and lineage parents. An audit export pins a query scope and policy, then returns events, authorized/redacted payloads, object proofs, checkpoint/Merkle proofs, signatures, schema dictionary, and a standalone verifier. APIs use producer idempotency keys and sequence reconciliation; deletion and hold are asynchronous, approval-gated jobs.

The application writes workflow state and an audit outbox row in one transaction. A relay sends canonical events to a regional append gateway, which authenticates workload/tenant, validates schema/sequence, durably writes replicated log segments, and acknowledges. Stream processors classify payloads, store encrypted objects, build the lineage graph and hot search, and reconcile event gaps. Periodic partition Merkle roots are signed by a protected key and anchored in a separately administered store.

The action records context manifest before the model call, model response hash, proposed tool call, authorization decision, human approval referencing that exact proposal digest, tool dispatch intent, and final receiver receipt. The payment service must implement a business idempotency key; timeout becomes OUTCOME_UNKNOWN and is reconciled by lookup. We do not claim a global transaction between audit log and payment, but the transactional outbox plus sequence/gap alarms gives detectable completeness.

Canonical serialization and hash chaining detect event modification, deletion, insertion, and reordering within a stream. Per-partition Merkle trees make compact inclusion proofs. Periodic roots are signed with a hardware-protected, rotated key and anchored to a separately controlled system or external transparency service, so a log admin cannot rewrite history and generate matching anchors. WORM retention and split duties add operational resistance.

This proves that exported bytes are included in an anchored ledger version and that the chain is structurally intact. It does not prove the producer told the truth or emitted every real-world action. Completeness needs producer sequence numbers, outbox backlog/gap reconciliation, cross-checks against model gateway/tool/payment logs, and independent control totals. Compromised producer keys can submit lies, so workload identity, key rotation/revocation, anomaly detection, and trusted timestamps matter.

A subject-resolution service verifies identity and maps subject tokens to source artifacts without exposing identity broadly. The lineage graph traverses from sources to chunks, prompts, model outputs, indexes, caches, evaluation items, exports, and backup retention sets. A versioned deletion plan classifies each node: delete bytes, crypto-erase a subject-specific key, redact/rederive a shared artifact, expire on backup schedule, preserve under hold, or retain minimal nonpersonal receipt metadata under legal basis.

Workers are idempotent on (deletion_request_id,node_id,lineage_version). Shared documents cannot be destroyed with one person’s key if others’ lawful data would disappear; we redact and rederive downstream objects under new digests, then tombstone old objects. Search indexes and caches delete immediately and are reconciled by scans. Backups are immutable: we record expiry, ensure deleted data is not restored to serving, and reapply the tombstone journal after restore. Model training requires a dataset manifest and model lineage; unlearning is a separate risk decision—never falsely attest deletion from a third-party model we cannot control.

The anchored Merkle root no longer matches the recomputed segment, and object hashes referenced by receipt events fail retrieval/integrity verification. The export returns an explicit incomplete status, proof of ledger inclusion, missing-object IDs/classifications, and incident reference—it never fabricates success. We preserve affected storage and access logs for forensics, revoke credentials, rotate keys, and compare replicas/object-lock histories.

To reduce the blast radius, no single operator controls storage deletion, signing keys, and anchors; payload deletion requires the policy workflow; WORM/object lock protects retained classes; signed roots are frequent enough to bound the rewrite window. Completeness dashboards continuously sample full proof construction, not only during an audit.

Hold and deletion operate on a serialized policy state per subject/artifact with monotonic version. A hold established by authorized legal personnel before a node’s destructive commit changes it to PRESERVE_RESTRICTED; later hold does not magically reverse already lawful deletion. Workers check the current policy version immediately before delete and write an append-only result. The audit shows the race order and authority.

Restore boots in quarantine, loads the latest signed tombstone/hold journal, and replays deletions before indexes or application reads are enabled. A deletion watermark in each derived store is compared with the global ledger; stale stores fail readiness. Continuous reconciliation looks for forbidden subject tokens/object digests. Held bytes use separate keys and tightly restricted purpose, and are deleted when the hold releases if no other basis remains.

Ingress partitions by tenant and time with quotas plus reserved compliance capacity; large payloads go directly to object storage so events stay bounded. We preserve minimal receipts before optional enrichments, shed search/index freshness rather than durable capture, and isolate bulk exports/deletion scans from live ingest. Finite retries and DLQs retain sequence visibility; producers spool locally with encrypted bounds when the gateway is unavailable and alert before overflow. If the transactional outbox or bounded spool cannot durably accept a high-impact action such as a payment or deployment, that action fails closed; lower-risk reads may defer emission but may never silently drop a receipt.

Tenant/region is enforced in identity, storage accounts or partitions, encryption contexts, queues, indexes, caches, lineage, and export workers. Anchors contain roots and signatures only, so integrity proof does not move regulated payloads across residency boundaries. ABAC binds case/purpose; export needs dual approval and field-level redaction; download URLs expire. Observe ingest gaps, sequence lag, event/object mismatch, Merkle checkpoint/anchor age, proof failures, search lag, deletion node states/SLA, backup expiry, hold access, cross-tenant denial, export completeness, cost/TB, and key health. DR replicates immutable segments/objects within approved regions and anchors under separate control; restore is gated by integrity verification plus tombstone replay.

First define the receipt schema and threat model with auditors/privacy/legal; instrument one low-risk workflow using outbox and sequence checks; run dual capture against existing logs; verify proof/export tooling independently; add payload classification and lineage; pilot deletion end to end including cache/index/backup restore; enable holds; then expand to tool/payment actions and more tenants. Keep completeness error budgets and periodic third-party verification.

The trade-off is audit fidelity versus data minimization. Storing full prompts forever makes replay easy but raises privacy, breach, and cost risk. I retain immutable minimal receipts and hashes under policy, encrypt payloads separately with precise lineage/retention, and make deletion itself an auditable event. Some regulated replay may be impossible after lawful deletion; the export must say that honestly.

I’ll summarize observable receipts, tamper evidence, completeness limits, and policy-driven lineage deletion.