8090 interview field guide · mock 20 of 20

Receipts Without Forever

Design a tamper-evident audit trail for AI actions that can still honor retention, legal holds, data residency, and deletion across every derived copy.

Invented rehearsal scenario · not a leaked question
30 minrealistic spoken transcript
12 Qsinterviewer prompts
50M actions/daymock workload
2 drillsdomain deep dives

↗ Read the evidence-grounded 8090 company research

The answer map

Do not memorize every box. Remember this path and rebuild the answer from it.

1 · ScopeWho uses it? What must it do?
2 · ScaleWhich numbers change the design?
3 · DataWhat must we save and never lose?
4 · Big boxesUse only a few major parts.
5 · Request pathFollow one example end to end.
6 · RiskFailures, security, rollout, recap.
Memory line: scope → numbers → data → big boxes → one request → risks.

Understand the problem

Restate the goal in plain English. Name the one thing the system must never get wrong.

InterviewerStarting question

Design an enterprise audit platform that records every AI action from prompt and retrieved context through model response, tool calls, permissions, human approvals, and final side effect. Auditors need reproducible exports and tamper evidence. Privacy teams need retention, legal hold, and deletion across derived data. Resolve the apparent conflict between “write-once” and “delete.”

Candidate
Start with the big pictureI will define what the system must protect before I choose technology.

“Cannot be silently changed” does not mean “keep every private byte forever.” I will separate two things. The first is a small write-once receipt: what happened, when, who approved it, and the fingerprints of its inputs and outputs. The second is the encrypted private content, which follows retention and deletion rules.

When data is deleted, we save a deletion receipt and remove every allowed copy we can trace. A legal hold can pause that removal for an approved reason. We record visible facts such as inputs, outputs, permissions, approvals, and tool results. We do not claim that hidden model reasoning is a reliable audit record.

Step 1 · Set the scope

Name the users, must-have features, speed and safety goals, and what is out of scope.

Interviewer

Scope actors, functions, service targets, and exclusions.

Candidate
Step 1 · Set the scopeI will name the users, the must-have features, the speed and safety goals, and what we will not build.

Applications and agents send receipts. Human approvers, customer admins, security responders, privacy officers, legal custodians, auditors, and reliability engineers use them. Version one receives receipts durably, joins the events for one action, searches and exports a timeline, proves whether stored records changed, applies retention and legal holds, and follows a deletion request through prompts, caches, indexes, exports, and backups.

For this mock, we record 50 million actions a day with 12 events each. The service should acknowledge 99 out of 100 writes within 200 milliseconds and make records searchable within 60 seconds. A normal deletion finishes within 30 days, and one action exports within two minutes. We do not store private model reasoning, prove that a model answer is true, or promise deletion from a third-party model we cannot control.

Actors

Agent/app producers, approvers, customer admins, security, privacy officers, legal custodians, auditors, SRE.

Functional

Receipts, payload vault, correlation, tamper proofs, search/export, retention, holds, data history deletion, backup cross-checking.

Correctness

Observable evidence only. Standard hashes. Sequence completeness. Deletion tombstones. No false unlearning/reproducibility claims.

NFR

50M actions/day. Ingest 99.99% p99 <200 ms; searchable <60 s; export <2 min; deletion <30 days; seven-year max.

Security

Customer/region isolation, separate encryption keys, signing with a protected device, dual-control export/hold, redaction, purpose-bound access policy.

Out of scope

Private chain-of-thought, truth proof, third-party model unlearning guarantee, audit store as workflow database.

Step 2 · Do the useful math

Estimate only what changes the design. After each number, say why it matters.

Interviewer

Use 50M actions/day, 12 events/action, 2 KB/event, 4:1 compression, seven years, and three replicas. Calculate.

Candidate
Step 2 · Use the numbersI will calculate only the numbers that can change the design, then explain why each result matters.

Fifty million actions times 12 events gives 600 million events a day, or about 6,944 a second on average. A ten-times burst is about 69,000 a second. At 2 KB per event, the uncompressed stream is 1.2 TB a day. With four-to-one compression and three copies, storage is about 900 GB a day, 328.5 TB a year, or 2.3 petabytes over seven years.

That size means we cannot keep every field in the fast search database. Recent records and a few safe search fields stay hot; older records move to cheaper locked storage and take longer to query. Large prompts and tool results live as separate encrypted objects identified by file fingerprint. Compliance receipts cannot be sampled, although ordinary performance logs can be.

Receipt ledger storage and ingestion

—events/day
—burst events/second
—logical TB/day
—physical PB over retention
Move a control to recalculate.
events/day = actions × events/action. Physical storage = events × KB ÷ compression × replicas × retention. Burst EPS = events/day ÷ 86,400 × burst

Steps 4–5 · APIs and data

Start with the few records the system cannot work without. Then add simple endpoints around them.

Interviewer

Define the records, keys, and APIs. What exactly is in an AI action receipt?

Candidate
Step 3 · Choose the dataI will keep the main records and APIs small. Each record has one clear job.

An Action groups everything that happened during one AI task. Each ReceiptEvent records its event type, actor, trusted arrival time, permission decision, model and tool versions, fingerprints of inputs and outputs, approvals, outside-system receipt, and a sequence number that only increases. It also stores the previous event’s fingerprint, so later edits break the chain.

Large private payloads are separate objects with data class, person or item IDs, region, encryption key, retention rule, and links to the copies created from them. An export is tied to one approved query and returns the events, permitted payloads, proof files, and a small verifier. Stable retry keys prevent duplicate receipts. Deletion and legal hold run as approved background jobs because they may touch many stores.

Records and how we find them

RecordMain key / retry keyFields we never overwriteHow we find it
Action(tenant_id, action_id)root/parent IDs, workflow/business key, class, region, start/end/outcometimeline; producer reconciliation
ReceiptEvent(tenant_id, time_partition, event_id)producer seq, type, actor, object/version hashes, prev hash, canonical hashappend scan; action timeline; integrity proof
PayloadObject(tenant_id, object_digest)classification, region, key ref, subjects/artifacts, retention, lineage parentsauthorized retrieval; deletion traversal
MerkleCheckpoint(tenant_id, partition_id, checkpoint_no)root, event range, signing key/version, external anchor receiptinclusion/consistency proof
LineageEdge(tenant_id, parent_digest, child_digest, transform_id)artifact types, transform version, subject tokens, discovered_atforward deletion; reverse impact
PolicyState(subject_or_artifact_id, policy_version)retention basis, holds, deletion request, authority, effective timeserialize deletion/hold decisions
DeletionNodeResult(request_id, node_id, lineage_version)action, worker, pre/post digest, status, proof, retrycompletion/reconciliation/attestation
AuditExport(tenant_id, export_id)scope/query hash, approval, redaction policy, checkpoint roots, bundle digestdownload; independent verification

Simple API

POST /v1/receipt-events                          Producer-Id + Sequence + Idempotency-Key
POST /v1/payloads/presign                         classification + region + subject tokens
GET  /v1/actions/{id}/timeline                    include sequence/completeness state
POST /v1/audit-exports                            scope + redaction policy + approvals
GET  /v1/audit-exports/{id}/bundle                signed bundle + standalone verifier
POST /v1/deletion-requests                        verified subject/artifact + legal basis
POST /v1/legal-holds                              custodian + scope + effective time
GET  /v1/deletion-requests/{id}/lineage-plan      versioned node actions
GET  /v1/checkpoints/{id}/proof?event_id=         inclusion + anchor chain
POST /v1/stores/{id}:reconcile-tombstones         deletion watermark

Steps 3 and 6 · Draw, then follow one request

Explain the system in a few parts. Then follow one real request from the user to the final result.

Interviewer

Walk through architecture and consistency. The model calls a payment tool after human approval.

Candidate
Steps 3 and 6 · Draw, then follow one requestI will explain the whole system in a few parts, then follow one request from start to finish.

I would draw four parts: receipt intake, the locked event log, encrypted payload storage, and search/export. The application saves its business change and an event-to-send in one database transaction. A relay sends that event to intake. Intake checks the caller and sequence, writes several durable copies, then replies. Background workers classify private payloads, store them separately, build search, and report missing sequence numbers.

For one payment action, we record the context versions, model response fingerprint, proposed tool call, permission check, human approval, send attempt, and the payment system’s receipt. If the payment call times out, its state is “outcome unknown”; we look it up using the same business key instead of blindly paying again. The audit log cannot share one transaction with the payment service, so saved events and gap alarms make missing records visible.

Receipts, proofs, and governed erasure Producersoutbox + sequencemodel · toolsAppend gatewayauth + schemadurable logProof planehash chainMerkle + anchorPayload + lineageencrypt · classifyderive + indexAudit / privacysearch · exportdelete · hold Independent controlsKMS/HSM · policy ledger · tombstone journalcompleteness reconciliation · DR · verifier

Prove inclusion and integrity—state the limit

Standard event hashes, chained ordering, Merkle checkpoints, split-control anchors, and producer cross-checking provide different guarantees.

Interviewer

Deep dive on cryptographic integrity. A database admin controls the log store. What can you prove, and what can’t you?

Candidate
Step 5 · Deep diveNow I will zoom in on the first hard part and use one concrete example.

Each event stores the fingerprint of the event before it. Changing a $100 payment to $1,000 changes that fingerprint and breaks the chain. For faster checks, we combine many fingerprints into a tree-shaped summary called a Merkle root. We regularly sign that root with a protected key and copy it to a separately controlled store. One log administrator therefore cannot rewrite the records and the checkpoint together.

This proves that the exported bytes were present in a particular saved log and have not been changed. It does not prove the original application told the truth or sent every event. For completeness, producers use sequence numbers, and we compare the audit stream with model, tool, and payment logs. That limit is important: cryptography detects tampering; it does not create truth.

Concrete example

An operator changes a logged payment amount from $100 to $1,000. Its fingerprint changes, which breaks the chain and no longer matches the separately stored checkpoint. We detect the edit. We still cannot prove the original producer told the truth.

Cryptographic receipt proof Canonical eventschema + bytesevent hashStream chainprevious hashorderingMerkle treepartition rootinclusion pathHSM signaturekey versiontrusted timeExternal anchorsplit controlrewrite detection Every transition records actor, input version, output version, reason, and timestamp.
MechanismProvesDoes not prove
Event hashBytes match digest.Producer told the truth.
Hash chainDetected change/order gap in one stream.Every real action was emitted.
Merkle proof + anchorEvent inclusion in externally fixed partition root.Business action was legitimate.
Sequence/outbox reconciliationDetectable producer emission gaps.Uninstrumented side effects.
Receiver receiptExternal system observed a business key/result.Global exactly-once semantics.

Delete private payloads, but keep a minimal receipt

Follow every copy, handle shared documents carefully, respect legal holds, and never promise deletion from a model you do not control.

Interviewer

Deep dive on deletion data history. A person asks to be erased. Their data is in prompts, chunks, embeddings, caches, eval sets, exports, and backups. Some saved items contain other people too.

Candidate
Step 6 · Deep diveNow I will zoom in on the second hard part and show how we know it works.

First we verify the person and find the source records that belong to them. Then we follow saved links from those records to search chunks, prompts, model outputs, indexes, caches, evaluation sets, exports, and backups. The deletion plan says what to do at each stop: remove bytes, destroy a person-specific encryption key, rebuild a shared document without that person, wait for a backup to expire, or preserve a restricted copy under legal hold.

For example, if one email appears in three search chunks and a shared report, we delete the chunks and rebuild the report without the email; we do not destroy other people’s lawful data. Stable job keys make retries safe. Search and caches are cleared quickly and then scanned for missed copies. After a backup restore, deletion markers run before users can search. We never claim that a third-party model forgot data when we cannot verify that.

Concrete example

A user’s email appears in a source file, three search chunks, a cache, and an evaluation set. The deletion plan follows that path. It removes private copies, rebuilds shared files without the email, clears indexes and caches, and records what must expire from backups later.

Policy-governed deletion Verify requestsubject tokensscope + basisFreeze lineageversioned planhold checkExecute nodesdelete / rederivecrypto-eraseReconcileindexes · cachesbackup watermarkAttestresults + gapssigned tombstone Every transition records actor, input version, output version, reason, and timestamp.
node_action ∈ {
  DELETE_BYTES, CRYPTO_ERASE, REDACT_AND_REDERIVE,
  DELETE_INDEX_ENTRY, EXPIRE_WITH_BACKUP,
  PRESERVE_RESTRICTED_UNDER_HOLD,
  RETAIN_MINIMAL_NONPERSONAL_RECEIPT
}

What if it breaks?

Protect the correct data first. Then contain the problem, recover, and explain what users see.

InterviewerFailure injection

A privileged operator deletes three payload objects and recomputes local segment hashes. Six hours later auditors ask for an export.

Candidate
When something breaksFirst protect correctness. Then contain the problem, recover, and prove what happened.

The anchored tree-shaped summary fingerprint (Merkle root) no longer matches the recomputed segment, and object hashes referenced by receipt events fail retrieval/integrity verification. The export returns an explicit incomplete status, proof of ledger inclusion, missing-object IDs/classifications, and incident reference—it never fabricates success. We preserve affected storage and access logs for forensics, revoke credentials, rotate keys, and compare replicas/object-lock histories.

To reduce the size of the impact, no single operator controls storage deletion, signing keys, and anchors. Payload deletion requires the policy workflow. Locked write-once storage/object lock protects retained classes. Signed roots are frequent enough to bound the rewrite window. Completeness dashboards continuously sample full proof construction, not only during an audit.

What if timing gets messy?

Retries, delayed messages, and two updates at once are normal. State the rule that keeps one clear result.

InterviewerFailure injection

A legal hold and deletion request race. Then a regional restore resurrects old search documents that should have been deleted.

Candidate
When two things raceI will make the winning rule explicit so retries and timing cannot create two truths.

Each person or item has one ordered policy record with a version number that only increases. A legal hold created before the final delete step changes the item to PRESERVE_RESTRICTED. A hold that arrives later cannot restore bytes already deleted lawfully. The worker reads the newest policy immediately before deletion and saves a permanent result showing which command won.

A restored backup starts in an isolated area. It loads the newest signed deletion-and-hold journal and applies those commands before search or application reads are allowed. Every rebuilt store reports which deletion version it has applied; a stale store cannot become ready. Held bytes use separate keys and purpose-limited access, then are deleted when the hold ends if no other legal reason remains.

Keep it safe and running

Cover overload, security, monitoring, and disaster recovery in that order.

Interviewer

Cover overload control, customer isolation, security, monitoring, exports, and DR.

Candidate
Run it safelyI will cover overload, security, monitoring, and recovery in that order.

Ingress partitions by customer and time with quotas plus reserved compliance capacity. Large payloads go directly to object storage so events stay bounded. We preserve minimal receipts before optional enrichments, shed search/index freshness rather than durable capture, and isolate bulk exports/deletion scans from live ingest. Finite retries and failed-job queue retain sequence visibility. Producers spool locally with encrypted bounds when the gateway is unavailable and alert before overflow.

Customer and region separation is enforced in identity, storage accounts or partitions, encryption contexts, queues, indexes, caches, data history, and export workers. Access policy binds case/purpose. Export needs dual approval and field-level redaction. Download URLs expire. Observe ingest gaps, sequence lag, event/object mismatch, Merkle checkpoint/anchor age, proof failures, search lag, deletion node states/SLA, backup expiry, hold access, cross-tenant denial, export completeness, cost/TB, and key health. DR replicates write-once segments/objects and anchors. Restore is gated by integrity verification plus tombstone replay.

Capture

Ingest EPS/latency/error, producer sequence gaps, local spool depth, database-backed event queue age, failed-job queue, event/payload mismatch.

Integrity

Checkpoint/anchor age, hash/proof failures, missing objects, key/signature verification, locked write-once storage policy violations.

Privacy

Deletion SLA/node failures, data history unknowns, index/cache cross-checking, backup freshness marker, hold access/release, resurrected tokens.

overload control

Reserved compliance ingest, per-tenant quota, bounded events, payload direct upload, search/export/deletion isolated from capture.

Isolation

Customer+region storage/keys/indexes/workers, access policy purpose/case, protected signing device split duties, dual-approved export, redacted telemetry.

DR

Replicate ledger/objects/anchors, verify before serve, replay latest tombstone/hold journal, rebuild indexes, prove restore.

Rollout and trade-offs

A credible production answer defines how it earns trust and how it retreats safely.

Interviewer

How do you roll this out and explain the immutability-versus-deletion trade-off?

Candidate
Ship in small stepsStart with the safest useful slice. Measure it, keep a quick rollback, and expand only after the gates pass.

First define the receipt schema and threat model with auditors/privacy/legal. Instrument one low-risk workflow using database-backed event queue and sequence checks. Run dual capture against existing logs. Verify proof/export tooling independently. Add payload classification and data history. Pilot deletion end to end including cache/index/backup restore. Enable holds. Then expand to tool/payment actions and more tenants. Keep completeness error budgets and periodic third-party verification.

The trade-off is audit fidelity versus data minimization. Storing full prompts forever makes replay easy but raises privacy, breach, and cost risk. I retain write-once minimal receipts and hashes under policy, encrypt payloads separately with precise data history/retention, and make deletion itself an auditable event. Some regulated replay may be impossible after lawful deletion. The export must say that honestly.

  1. Agree on observable receipt schema, retention basis, and threat model.
  2. Instrument one workflow with database-backed event queue and producer sequences.
  3. Dual-capture and cross-check against model/tool/receiver logs.
  4. Verify hash/Merkle/export bundle with an independent tool/team.
  5. Add payload classification, subject tokens, and data history.
  6. Pilot deletion through caches, indexes, shared saved items, and backup restore.
  7. Enable legal hold race tests and dual-control exports.
  8. Expand by action class only when completeness, proof, privacy, cost, and service target gates pass.

One-minute spoken recap

Practice this synthesis until it sounds conversational rather than memorized.

Interviewer

One-minute close.

Candidate
Close with the mapProblem, scale, big boxes, hardest risk, and rollout.

I’ll summarize observable receipts, tamper evidence, completeness limits, and policy-driven data history deletion.

“Cannot be silently changed” does not mean “keep every private byte forever.” Applications save ordered receipt events for context, model, permission, approval, tools, and final outside effects. Regional gateways store them durably. Fingerprint chains and signed Merkle checkpoints expose later edits, while sequence numbers and comparisons with payment or tool logs expose missing events. Sensitive payloads stay separate and encrypted. A trace of every derived copy drives deletion, handles shared documents by rebuilding them, respects legal holds, and reapplies deletion markers before a restored backup can serve. Exports include proofs and clearly disclose anything missing or lawfully deleted.

Reference shelf

Definitions, traps, and the final checklist stay outside the timed mock.

Canonical serialization
One deterministic byte representation used to compute stable hashes.
Cryptographic erasure
Destroying a data-encryption key so ciphertext becomes unreadable.
Hash chain
Each event commits to the previous event hash, exposing mutation or reordering.
Legal hold
Authorized preservation that suspends normal deletion for defined material and purpose.
Lineage graph
Edges from source artifacts to every derived artifact/transformation.
Merkle tree
Hash tree enabling compact proof that an item is included in a fixed root.
Observable receipt
Record of inputs, versions, decisions, tool calls, approvals, outputs, and effects—not private reasoning.
Outbox reconciliation
Checking transactionally captured producer intents against emitted ledger sequences.
Redact and rederive
Remove one subject from a shared object and regenerate descendants under new hashes.
Tombstone journal
Append-only record of material that must remain deleted after rebuild/restore.
Transparency anchor
A root stored under independent control so history cannot be silently re-rooted.
Unlearning
Attempt to remove training influence from a model; distinct from deleting source/evaluation artifacts.
  1. Equating write-once with “retain all prompts forever.”
  2. Storing hidden chain-of-thought as a trustworthy audit fact.
  3. Hashing events without externally anchoring the root.
  4. Claiming hashes prove the producer told the truth.
  5. Ignoring missing producer events and uninstrumented side effects.
  6. Putting full sensitive payloads in the hot searchable ledger.
  7. Deleting a shared saved item key and erasing other people’s lawful data.
  8. Promising third-party foundation-model unlearning without control/evidence.
  9. Restoring a backup directly to service before tombstone replay.
  10. Letting one admin control storage, signing keys, and anchors.
  11. Blindly retrying a payment after an unknown timeout.
  12. Returning a “complete” audit bundle when payloads/proofs are missing.
  • Clarified actors, authority, business harm, and out-of-scope.
  • Did correct BOTEC and named the variable that changes the architecture.
  • Defined stable IDs, versions, access paths, and retry keys.
  • Separated source of truth state from derived indexes and model output.
  • Explained consistency, retries, overload control, and unknown outcomes.
  • Revised the design after both failure injections.
  • Covered customer isolation, secrets, least privilege, deletion, and audit.
  • Named golden signals, domain quality metrics, rollout gates, and rollback.