Design an enterprise audit platform that records every AI action from prompt and retrieved context through model response, tool calls, permissions, human approvals, and final side effect. Auditors need reproducible exports and tamper evidence. Privacy teams need retention, legal hold, and deletion across derived data. Resolve the apparent conflict between “write-once” and “delete.”
Receipts Without Forever
Design a tamper-evident audit trail for AI actions that can still honor retention, legal holds, data residency, and deletion across every derived copy.
The answer map
Do not memorize every box. Remember this path and rebuild the answer from it.
Understand the problem
Restate the goal in plain English. Name the one thing the system must never get wrong.
“Cannot be silently changed” does not mean “keep every private byte forever.” I will separate two things. The first is a small write-once receipt: what happened, when, who approved it, and the fingerprints of its inputs and outputs. The second is the encrypted private content, which follows retention and deletion rules.
When data is deleted, we save a deletion receipt and remove every allowed copy we can trace. A legal hold can pause that removal for an approved reason. We record visible facts such as inputs, outputs, permissions, approvals, and tool results. We do not claim that hidden model reasoning is a reliable audit record.
Step 1 · Set the scope
Name the users, must-have features, speed and safety goals, and what is out of scope.
Scope actors, functions, service targets, and exclusions.
Applications and agents send receipts. Human approvers, customer admins, security responders, privacy officers, legal custodians, auditors, and reliability engineers use them. Version one receives receipts durably, joins the events for one action, searches and exports a timeline, proves whether stored records changed, applies retention and legal holds, and follows a deletion request through prompts, caches, indexes, exports, and backups.
For this mock, we record 50 million actions a day with 12 events each. The service should acknowledge 99 out of 100 writes within 200 milliseconds and make records searchable within 60 seconds. A normal deletion finishes within 30 days, and one action exports within two minutes. We do not store private model reasoning, prove that a model answer is true, or promise deletion from a third-party model we cannot control.
Agent/app producers, approvers, customer admins, security, privacy officers, legal custodians, auditors, SRE.
Receipts, payload vault, correlation, tamper proofs, search/export, retention, holds, data history deletion, backup cross-checking.
Observable evidence only. Standard hashes. Sequence completeness. Deletion tombstones. No false unlearning/reproducibility claims.
50M actions/day. Ingest 99.99% p99 <200 ms; searchable <60 s; export <2 min; deletion <30 days; seven-year max.
Customer/region isolation, separate encryption keys, signing with a protected device, dual-control export/hold, redaction, purpose-bound access policy.
Private chain-of-thought, truth proof, third-party model unlearning guarantee, audit store as workflow database.
Step 2 · Do the useful math
Estimate only what changes the design. After each number, say why it matters.
Use 50M actions/day, 12 events/action, 2 KB/event, 4:1 compression, seven years, and three replicas. Calculate.
Fifty million actions times 12 events gives 600 million events a day, or about 6,944 a second on average. A ten-times burst is about 69,000 a second. At 2 KB per event, the uncompressed stream is 1.2 TB a day. With four-to-one compression and three copies, storage is about 900 GB a day, 328.5 TB a year, or 2.3 petabytes over seven years.
That size means we cannot keep every field in the fast search database. Recent records and a few safe search fields stay hot; older records move to cheaper locked storage and take longer to query. Large prompts and tool results live as separate encrypted objects identified by file fingerprint. Compliance receipts cannot be sampled, although ordinary performance logs can be.
Receipt ledger storage and ingestion
Steps 4–5 · APIs and data
Start with the few records the system cannot work without. Then add simple endpoints around them.
Define the records, keys, and APIs. What exactly is in an AI action receipt?
An Action groups everything that happened during one AI task. Each ReceiptEvent records its event type, actor, trusted arrival time, permission decision, model and tool versions, fingerprints of inputs and outputs, approvals, outside-system receipt, and a sequence number that only increases. It also stores the previous event’s fingerprint, so later edits break the chain.
Large private payloads are separate objects with data class, person or item IDs, region, encryption key, retention rule, and links to the copies created from them. An export is tied to one approved query and returns the events, permitted payloads, proof files, and a small verifier. Stable retry keys prevent duplicate receipts. Deletion and legal hold run as approved background jobs because they may touch many stores.
Records and how we find them
| Record | Main key / retry key | Fields we never overwrite | How we find it |
|---|---|---|---|
Action | (tenant_id, action_id) | root/parent IDs, workflow/business key, class, region, start/end/outcome | timeline; producer reconciliation |
ReceiptEvent | (tenant_id, time_partition, event_id) | producer seq, type, actor, object/version hashes, prev hash, canonical hash | append scan; action timeline; integrity proof |
PayloadObject | (tenant_id, object_digest) | classification, region, key ref, subjects/artifacts, retention, lineage parents | authorized retrieval; deletion traversal |
MerkleCheckpoint | (tenant_id, partition_id, checkpoint_no) | root, event range, signing key/version, external anchor receipt | inclusion/consistency proof |
LineageEdge | (tenant_id, parent_digest, child_digest, transform_id) | artifact types, transform version, subject tokens, discovered_at | forward deletion; reverse impact |
PolicyState | (subject_or_artifact_id, policy_version) | retention basis, holds, deletion request, authority, effective time | serialize deletion/hold decisions |
DeletionNodeResult | (request_id, node_id, lineage_version) | action, worker, pre/post digest, status, proof, retry | completion/reconciliation/attestation |
AuditExport | (tenant_id, export_id) | scope/query hash, approval, redaction policy, checkpoint roots, bundle digest | download; independent verification |
Simple API
POST /v1/receipt-events Producer-Id + Sequence + Idempotency-Key
POST /v1/payloads/presign classification + region + subject tokens
GET /v1/actions/{id}/timeline include sequence/completeness state
POST /v1/audit-exports scope + redaction policy + approvals
GET /v1/audit-exports/{id}/bundle signed bundle + standalone verifier
POST /v1/deletion-requests verified subject/artifact + legal basis
POST /v1/legal-holds custodian + scope + effective time
GET /v1/deletion-requests/{id}/lineage-plan versioned node actions
GET /v1/checkpoints/{id}/proof?event_id= inclusion + anchor chain
POST /v1/stores/{id}:reconcile-tombstones deletion watermarkSteps 3 and 6 · Draw, then follow one request
Explain the system in a few parts. Then follow one real request from the user to the final result.
Walk through architecture and consistency. The model calls a payment tool after human approval.
I would draw four parts: receipt intake, the locked event log, encrypted payload storage, and search/export. The application saves its business change and an event-to-send in one database transaction. A relay sends that event to intake. Intake checks the caller and sequence, writes several durable copies, then replies. Background workers classify private payloads, store them separately, build search, and report missing sequence numbers.
For one payment action, we record the context versions, model response fingerprint, proposed tool call, permission check, human approval, send attempt, and the payment system’s receipt. If the payment call times out, its state is “outcome unknown”; we look it up using the same business key instead of blindly paying again. The audit log cannot share one transaction with the payment service, so saved events and gap alarms make missing records visible.
Prove inclusion and integrity—state the limit
Standard event hashes, chained ordering, Merkle checkpoints, split-control anchors, and producer cross-checking provide different guarantees.
Deep dive on cryptographic integrity. A database admin controls the log store. What can you prove, and what can’t you?
Each event stores the fingerprint of the event before it. Changing a $100 payment to $1,000 changes that fingerprint and breaks the chain. For faster checks, we combine many fingerprints into a tree-shaped summary called a Merkle root. We regularly sign that root with a protected key and copy it to a separately controlled store. One log administrator therefore cannot rewrite the records and the checkpoint together.
This proves that the exported bytes were present in a particular saved log and have not been changed. It does not prove the original application told the truth or sent every event. For completeness, producers use sequence numbers, and we compare the audit stream with model, tool, and payment logs. That limit is important: cryptography detects tampering; it does not create truth.
An operator changes a logged payment amount from $100 to $1,000. Its fingerprint changes, which breaks the chain and no longer matches the separately stored checkpoint. We detect the edit. We still cannot prove the original producer told the truth.
| Mechanism | Proves | Does not prove |
|---|---|---|
| Event hash | Bytes match digest. | Producer told the truth. |
| Hash chain | Detected change/order gap in one stream. | Every real action was emitted. |
| Merkle proof + anchor | Event inclusion in externally fixed partition root. | Business action was legitimate. |
| Sequence/outbox reconciliation | Detectable producer emission gaps. | Uninstrumented side effects. |
| Receiver receipt | External system observed a business key/result. | Global exactly-once semantics. |
Delete private payloads, but keep a minimal receipt
Follow every copy, handle shared documents carefully, respect legal holds, and never promise deletion from a model you do not control.
Deep dive on deletion data history. A person asks to be erased. Their data is in prompts, chunks, embeddings, caches, eval sets, exports, and backups. Some saved items contain other people too.
First we verify the person and find the source records that belong to them. Then we follow saved links from those records to search chunks, prompts, model outputs, indexes, caches, evaluation sets, exports, and backups. The deletion plan says what to do at each stop: remove bytes, destroy a person-specific encryption key, rebuild a shared document without that person, wait for a backup to expire, or preserve a restricted copy under legal hold.
For example, if one email appears in three search chunks and a shared report, we delete the chunks and rebuild the report without the email; we do not destroy other people’s lawful data. Stable job keys make retries safe. Search and caches are cleared quickly and then scanned for missed copies. After a backup restore, deletion markers run before users can search. We never claim that a third-party model forgot data when we cannot verify that.
A user’s email appears in a source file, three search chunks, a cache, and an evaluation set. The deletion plan follows that path. It removes private copies, rebuilds shared files without the email, clears indexes and caches, and records what must expire from backups later.
node_action ∈ {
DELETE_BYTES, CRYPTO_ERASE, REDACT_AND_REDERIVE,
DELETE_INDEX_ENTRY, EXPIRE_WITH_BACKUP,
PRESERVE_RESTRICTED_UNDER_HOLD,
RETAIN_MINIMAL_NONPERSONAL_RECEIPT
}What if it breaks?
Protect the correct data first. Then contain the problem, recover, and explain what users see.
A privileged operator deletes three payload objects and recomputes local segment hashes. Six hours later auditors ask for an export.
The anchored tree-shaped summary fingerprint (Merkle root) no longer matches the recomputed segment, and object hashes referenced by receipt events fail retrieval/integrity verification. The export returns an explicit incomplete status, proof of ledger inclusion, missing-object IDs/classifications, and incident reference—it never fabricates success. We preserve affected storage and access logs for forensics, revoke credentials, rotate keys, and compare replicas/object-lock histories.
To reduce the size of the impact, no single operator controls storage deletion, signing keys, and anchors. Payload deletion requires the policy workflow. Locked write-once storage/object lock protects retained classes. Signed roots are frequent enough to bound the rewrite window. Completeness dashboards continuously sample full proof construction, not only during an audit.
What if timing gets messy?
Retries, delayed messages, and two updates at once are normal. State the rule that keeps one clear result.
A legal hold and deletion request race. Then a regional restore resurrects old search documents that should have been deleted.
Each person or item has one ordered policy record with a version number that only increases. A legal hold created before the final delete step changes the item to PRESERVE_RESTRICTED. A hold that arrives later cannot restore bytes already deleted lawfully. The worker reads the newest policy immediately before deletion and saves a permanent result showing which command won.
A restored backup starts in an isolated area. It loads the newest signed deletion-and-hold journal and applies those commands before search or application reads are allowed. Every rebuilt store reports which deletion version it has applied; a stale store cannot become ready. Held bytes use separate keys and purpose-limited access, then are deleted when the hold ends if no other legal reason remains.
Keep it safe and running
Cover overload, security, monitoring, and disaster recovery in that order.
Cover overload control, customer isolation, security, monitoring, exports, and DR.
Ingress partitions by customer and time with quotas plus reserved compliance capacity. Large payloads go directly to object storage so events stay bounded. We preserve minimal receipts before optional enrichments, shed search/index freshness rather than durable capture, and isolate bulk exports/deletion scans from live ingest. Finite retries and failed-job queue retain sequence visibility. Producers spool locally with encrypted bounds when the gateway is unavailable and alert before overflow.
Customer and region separation is enforced in identity, storage accounts or partitions, encryption contexts, queues, indexes, caches, data history, and export workers. Access policy binds case/purpose. Export needs dual approval and field-level redaction. Download URLs expire. Observe ingest gaps, sequence lag, event/object mismatch, Merkle checkpoint/anchor age, proof failures, search lag, deletion node states/SLA, backup expiry, hold access, cross-tenant denial, export completeness, cost/TB, and key health. DR replicates write-once segments/objects and anchors. Restore is gated by integrity verification plus tombstone replay.
Ingest EPS/latency/error, producer sequence gaps, local spool depth, database-backed event queue age, failed-job queue, event/payload mismatch.
Checkpoint/anchor age, hash/proof failures, missing objects, key/signature verification, locked write-once storage policy violations.
Deletion SLA/node failures, data history unknowns, index/cache cross-checking, backup freshness marker, hold access/release, resurrected tokens.
Reserved compliance ingest, per-tenant quota, bounded events, payload direct upload, search/export/deletion isolated from capture.
Customer+region storage/keys/indexes/workers, access policy purpose/case, protected signing device split duties, dual-approved export, redacted telemetry.
Replicate ledger/objects/anchors, verify before serve, replay latest tombstone/hold journal, rebuild indexes, prove restore.
Rollout and trade-offs
A credible production answer defines how it earns trust and how it retreats safely.
How do you roll this out and explain the immutability-versus-deletion trade-off?
First define the receipt schema and threat model with auditors/privacy/legal. Instrument one low-risk workflow using database-backed event queue and sequence checks. Run dual capture against existing logs. Verify proof/export tooling independently. Add payload classification and data history. Pilot deletion end to end including cache/index/backup restore. Enable holds. Then expand to tool/payment actions and more tenants. Keep completeness error budgets and periodic third-party verification.
The trade-off is audit fidelity versus data minimization. Storing full prompts forever makes replay easy but raises privacy, breach, and cost risk. I retain write-once minimal receipts and hashes under policy, encrypt payloads separately with precise data history/retention, and make deletion itself an auditable event. Some regulated replay may be impossible after lawful deletion. The export must say that honestly.
- Agree on observable receipt schema, retention basis, and threat model.
- Instrument one workflow with database-backed event queue and producer sequences.
- Dual-capture and cross-check against model/tool/receiver logs.
- Verify hash/Merkle/export bundle with an independent tool/team.
- Add payload classification, subject tokens, and data history.
- Pilot deletion through caches, indexes, shared saved items, and backup restore.
- Enable legal hold race tests and dual-control exports.
- Expand by action class only when completeness, proof, privacy, cost, and service target gates pass.
One-minute spoken recap
Practice this synthesis until it sounds conversational rather than memorized.
One-minute close.
I’ll summarize observable receipts, tamper evidence, completeness limits, and policy-driven data history deletion.
Reference shelf
Definitions, traps, and the final checklist stay outside the timed mock.
- Canonical serialization
- One deterministic byte representation used to compute stable hashes.
- Cryptographic erasure
- Destroying a data-encryption key so ciphertext becomes unreadable.
- Hash chain
- Each event commits to the previous event hash, exposing mutation or reordering.
- Legal hold
- Authorized preservation that suspends normal deletion for defined material and purpose.
- Lineage graph
- Edges from source artifacts to every derived artifact/transformation.
- Merkle tree
- Hash tree enabling compact proof that an item is included in a fixed root.
- Observable receipt
- Record of inputs, versions, decisions, tool calls, approvals, outputs, and effects—not private reasoning.
- Outbox reconciliation
- Checking transactionally captured producer intents against emitted ledger sequences.
- Redact and rederive
- Remove one subject from a shared object and regenerate descendants under new hashes.
- Tombstone journal
- Append-only record of material that must remain deleted after rebuild/restore.
- Transparency anchor
- A root stored under independent control so history cannot be silently re-rooted.
- Unlearning
- Attempt to remove training influence from a model; distinct from deleting source/evaluation artifacts.
- Equating write-once with “retain all prompts forever.”
- Storing hidden chain-of-thought as a trustworthy audit fact.
- Hashing events without externally anchoring the root.
- Claiming hashes prove the producer told the truth.
- Ignoring missing producer events and uninstrumented side effects.
- Putting full sensitive payloads in the hot searchable ledger.
- Deleting a shared saved item key and erasing other people’s lawful data.
- Promising third-party foundation-model unlearning without control/evidence.
- Restoring a backup directly to service before tombstone replay.
- Letting one admin control storage, signing keys, and anchors.
- Blindly retrying a payment after an unknown timeout.
- Returning a “complete” audit bundle when payloads/proofs are missing.
- Clarified actors, authority, business harm, and out-of-scope.
- Did correct BOTEC and named the variable that changes the architecture.
- Defined stable IDs, versions, access paths, and retry keys.
- Separated source of truth state from derived indexes and model output.
- Explained consistency, retries, overload control, and unknown outcomes.
- Revised the design after both failure injections.
- Covered customer isolation, secrets, least privilege, deletion, and audit.
- Named golden signals, domain quality metrics, rollout gates, and rollback.