Owns identity, policy, workflow state, budgets, approvals, leases, and audit. It decides what may happen.
System design field manual · Mock 01
Design an auditable AI software factory.
A realistic interview transcript and visual reference for a multi-tenant control plane that connects intent to code, runs durable agents, and can explain every consequential action.
- Candidate
- You · Staff-level full-stack / distributed systems
- Format
- Live architecture review
- Clock
- 30 minutes · 9 timed acts
- Deep dives
- Versioned provenance + durable multi-model execution
- Difficulty
- Enterprise critical
00Before the clock
Definitions and mental model
Forge is not “ChatGPT that writes code.” It is an authority system wrapped around fallible workers.
Runs untrusted or failure-prone work: repository parsing, model calls, tools, tests, and sandboxed agents.
A typed, version-specific claim that one thing derives from, implements, tests, contradicts, or supersedes another.
Logical work survives process, machine, provider, and region failures because progress is checkpointed outside the worker.
The central invariant
Models propose; policy and humans authorize. An agent may create a suggestion or pull request. It never silently rewrites published truth.
The storage split
Immutable versions and audit events preserve history. Transactional current views make the product usable. Derived indexes are rebuildable.
The delivery promise
A release pins exact versions of requirement, blueprint, commit, tests, policy, model inputs, approvals, and receipts.
Interview posture
Drive from consequences to invariants to data. Technology names come after you can state what must never happen.
0100:00–03:00
Opening and scope
Listen, restate, and spend your first questions buying down ambiguity.
Interviewer · starting question Design Forge: a multi-tenant system where product teams create requirements and blueprints, turn them into dependency-aware work orders, run coding agents for minutes or hours, connect changes to repositories and tests, detect drift, and export a complete audit trail. Start with scope and invariants, estimate load, then show data, APIs, architecture, provenance, and durable execution.
Interviewer
Assume 250 enterprises, 200,000 users, 20,000 projects, 80,000 repositories, 500 million provenance edges, 15,000 webhooks per second at peak, and 5,000 concurrent agent runs. Where do you start?
Candidate
Candidate · clear top-down answerI would start by splitting the product into three jobs: store approved plans, build searchable views from them and the code, and run long AI coding jobs safely. Then I would ask what must be reviewed, what an agent may change, and what proof we need after a run.
Interviewer
Published versions are immutable. Drift is advisory until a customer makes it a release policy. Agents may read, analyze, run tests, and open a PR; merging and deployment require human approval. Audit must explain and verify an action, not regenerate identical tokens.
Candidate
Candidate · clear top-down answerMy main rule is simple: every important output must say exactly which requirements, code version, policy, model, tool calls, and approvals produced it. Search can be a little behind. Approved plans and final decisions cannot be vague.
Decision
In scope
Boundary / why
Authoring
Structured requirements, blueprints, versions, comments, suggestions, publication
Not a general IDE
Execution
Work-order DAG, agent scheduling, checkpoints, tools, tests, PR creation
Foundation model training excluded
Intelligence
Incremental code index, typed links, impact and drift findings
Parsers are plugins with versioned contracts
Governance
Tenant policy, approvals, budgets, audit bundle, deletion lineage
External IdP and SCM remain systems of record
0203:00–06:00
Turn requirements into clear rules
Translate product verbs into rules the architecture must protect.
Interviewer
What are your functional and nonfunctional priorities?
Candidate
Candidate · clear top-down answerThe must-have actions are editing and publishing plans, creating work orders, running agents, reviewing results, and keeping an audit history. These small control records need strong database checks. Search, code maps, notifications, and large impact reports can update in the background.
Interviewer
Can a product manager publish while a 100,000-edge impact computation is running?
Candidate
Candidate · clear top-down answerPublishing should not wait for a huge impact scan. Save the new fixed version and a durable 'analyze this' job together, return success, and let a worker build the report. A release can wait for that report later if policy requires it.
Authority
A published version never changes. Corrections create a successor. A release pins exact versions, never “latest.”
Isolation
Every access path—including search, cache, log, prompt, and eval—is tenant- and policy-scoped.
Execution
One logical run can have many attempts; only the current fenced lease may advance state.
Evidence
Auditable means observable evidence and version manifests, not hidden chain-of-thought or guaranteed identical generation.
| Need | Target | Design consequence | Degrade safely as |
|---|---|---|---|
| Authoritative writes | P95 < 300 ms; 99.95% | Regional transactional authority, optimistic concurrency, outbox | Read-only with explicit status if quorum is unavailable |
| Collaboration | Remote update < 250 ms | Ephemeral presence + operation stream; publish remains separate | Local draft and reconnect conflict UI |
| Drift | First finding < 5 min | Priority queue, incremental index, partial generation markers | Stale badge; block only if policy says so |
| Agent runs | 1 min–8 hr; 5,000 active | Leases, heartbeats, checkpoints, cancellation epoch, quotas | Continue without live stream; pause before privileged step |
| Disaster recovery | RPO < 1 min; RTO < 30 min | Cross-region log replication, fenced failover, rehearsed restore | No dual writers; preserve safety over write availability |
| Audit | 7 years | Append-oriented events, WORM export, integrity manifests, legal hold | Audit querying can lag; recording cannot silently drop |
0306:00–09:00
Rough scale math
Use estimates to find the expensive paths: not user CRUD, but write amplification, fan-out, indexing, and long-lived work.
Interviewer
Give me enough math to defend your partitions and asynchronous boundaries.
Candidate
Candidate · clear top-down answerThe user traffic is manageable, but one action creates several background jobs. At 5,000 writes per second and about eight follow-up events each, the event system sees roughly 40,000 events per second. This is why I separate fast control writes from heavy background work.
Write amplification
5,000 writes/s × 8 events ≈ 40,000 events/sPartition by tenant/project and event family; avoid one global ordering requirement.
Webhook burst
15,000/s × 2 KB ≈ 30 MB/sDurable ingress is cheap; fetching and parsing monorepos is not. Coalesce before work.
Run heartbeat
5,000 ÷ 30 sec ≈ 167/sThe scheduler bottleneck is fairness and side effects, not raw heartbeat throughput.
Audit & provenance capacity calculator
Move the assumptions. The formulas update locally—no data leaves this page.
Planning model only, using binary capacity units: compression, hot/cold tiers, index shape, legal holds, and tenant skew change real capacity. The point is to expose assumptions.
Workload shape matters more than averages
Bars compare architectural pressure, not common units. Fan-out and tenant skew drive queues, cache policy, and work limits.
0409:00–12:00
Simple data model and APIs
Stable logical identity, immutable versions, typed relationships, and idempotent commands form the spine.
Interviewer
Show me the minimum schema. Be precise about identity and versioning.
Candidate
Candidate · clear top-down answerThe main records are plan, fixed plan version, published release, link between two exact versions, work order, agent run, worker attempt, approval, and tool receipt. For example, a test must point to requirement version 8, not merely 'the latest requirement.'
Core entity model
Stable ID ≠ immutable version| Record | Primary identity | Critical invariant | Main access pattern |
|---|---|---|---|
ArtifactVersion | (org_id, project_id, version_id) | Content/hash immutable after commit | Resolve a release or compare parent/successor |
ProvenanceEdge | (tenant_partition, edge_id) | Both endpoints are exact versions/spans | Bounded inbound/outbound traversal by type |
WorkOrderDependency | (project_id, from_id, to_id) | No cycle; readiness computed from terminal prerequisites | Ready queue, dependency update |
AgentRun | (org_id, run_id) | One logical billable intent; monotonic state version | Status, cancel, resume, audit manifest |
RunAttempt | (run_id, attempt_no) | Only fenced current lease may commit progress | Recovery, debugging, cost reconciliation |
AuditEvent | (tenant_shard, time_bucket, event_id) | Append-only; hash-chain/checkpoint integrity | Release/feature export and incident timeline |
Publish an artifact version
POST /v1/projects/prj_7/artifacts/req_42/versions If-Match: "v7:sha256:9f..." Idempotency-Key: "9da7..." { "base_version_id": "av_v7", "content": { "acceptance": [ ... ] }, "operation": "PUBLISH" } → 201 version av_v8 + analysis_generation ag_91 → 412 STALE_BASE with current etag + structured diff
Launch one logical run
POST /v1/work-orders/wo_81/agent-runs Idempotency-Key: "launch:wo_81:3" { "context_manifest": "cm_773", "capability_profile": "code-pr-only", "budget": { "usd_micros": 25000000 }, "provider_policy": ["private-a", "fallback-b"] } → 202 run ar_55; duplicate key returns ar_55 → 409 STALE_CONTEXT / 402 BUDGET_NOT_RESERVED
API principle
Retrying a command returns the original logical result. It must not publish a second version, reserve budget twice, or launch a second run.
0512:00–16:00
High-level architecture
Keep synchronous authority small; let every expensive derivative be replayable and visibly stale.
Interviewer
Walk one requirement edit to a tested pull request.
Candidate
Candidate · clear top-down answerAt a high level there are three large parts. The control service stores approved truth. Background workers build search and impact views. The run service schedules isolated agent workers. A message stream connects them without making the user wait.
Forge logical architecture
Solid = authority · dashed = asynchronousTransactional outbox
Truth and “work must happen” commit together. Consumers are at-least-once and idempotent by event ID + analysis generation.
Immutable commit keys
Repository analysis is keyed by provider, repo, and commit SHA—not webhook order or mutable branch name.
Visible freshness
Search results and drift findings display index generation, source commit, coverage, and lag. “Partial” is a first-class state.
0616:00–20:00
Deep dive: follow a change to everything it affects
The hard part is not drawing a graph. It is defining what an edge means when time, evidence, and authority change.
Interviewer
REQ-42 v7 is published. Forty dependent work orders are active and three agents opened PRs. A PM publishes v8 with a stricter safety constraint. What happens?
Candidate
Candidate · clear top-down answerIf requirement version 8 changes a safety limit, version 7 stays unchanged. We find code, tests, work orders, and open pull requests linked to version 7. Running jobs keep their old input until a policy check pauses them before a risky tool call.
Interviewer
Why not put everything in a graph database and run the closure synchronously?
Candidate
Candidate · clear top-down answerA graph database can make link-walking fast, but it does not decide which version is true. I would store the trusted link record in the main database and rebuild the graph view when needed. Limit every scan by release, link type, depth, and item count.
REQ-42 v7Safety criterion AC-9 · published
BP-11 v4Implements v7 · reviewed architecture
WO-81Pinned v7 · now stale pending review
RUN-55Context manifest cm-773 · attempt 2
PR #184Commit 2a91 · check marks stale context
TEST-AC9Passes old criterion; insufficient for v8
RELEASE R-12Pins v7 until explicit revalidation
An edge is a claim, not a pointer
(source_version, source_span, target_version, target_span, type, asserter, evidence, confidence, policy, valid_from, superseded_by)
A model-suggested “implements” edge and a human-approved one are different facts even if endpoints match.
A release is a closed manifest
Artifact versions, commits, test runs, policies, approvals, tool receipts, and analysis generations are pinned and hash-addressed.
Never resolve “latest” during audit export; it changes underneath you.
| Edge type | Meaning | Who may assert | Impact direction | Release behavior |
|---|---|---|---|---|
DERIVED_FROM | Output used this exact source/span | Pipeline or human | Source change → output stale candidate | Evidence required |
IMPLEMENTS | Code/work realizes a requirement | Agent suggests; human/policy approves | Both directions | Approved edge required |
TESTS | Test exercises acceptance criterion | QA/agent proposal | Criterion → test | Passing pinned run required |
CONTRADICTS | Evidence conflicts with target claim | Analyzer or reviewer | Finding only | May block by policy |
SUPERSEDES | New version/decision replaces old | Authoritative workflow | Temporal navigation | Old remains auditable |
MENTIONS | Weak reference without implementation claim | Parser/search | Usually no invalidation | Never enough alone |
Consistency is chosen per invariant
Operation
Strong / transactional
Eventual / derived
User-visible guard
Publish version
✓ base etag + immutable write
—
Conflict diff on stale base
Approval / release pin
✓ policy + decision
—
Signed decision history
Impact closure
Seed event only
✓ generation traversal
Progress, generation, incomplete flag
Search / embeddings
ACL source of truth
✓ index refresh
Filter at query + post-filter; freshness
Realtime text
Publish boundary
✓ operation propagation
Draft/published states separate
Metrics
Billing ledger event
✓ dashboard aggregate
Estimated vs finalized cost
Collaboration trap
OT or CRDT may merge text operations, but it must not merge approval state, publish two conflicting base versions, or turn an agent suggestion into authoritative content. Semantic workflow remains transactional.
0720:00–24:00
Deep dive: keep long agent jobs safe
A run is a workflow with nondeterministic compute and ambiguous side effects—not a long HTTP request.
Interviewer
My laptop closes while an eight-hour agent is running. Then its worker dies. How does it continue safely?
Candidate
Candidate · clear top-down answerA long agent job belongs to the server, not the user's laptop. A worker receives a timed lease, saves progress at safe steps, and sends heartbeats. If it dies, another worker resumes from the last saved step; the old worker loses permission to commit late results.
Interviewer
The agent called an external ticket API. The ticket may have been created, but the response was lost. The retry is about to call again. Give me “exactly once.”
Candidate
Candidate · clear top-down answerWe cannot promise exactly one external action unless the external service helps. Before a tool call, save one stable action key. After a timeout, look up that key before retrying. If we cannot know the result, stop and ask a person instead of risking a duplicate harmful action.
Interviewer
The preferred model provider is down. Can you fail over?
Candidate
Candidate · clear top-down answerA model retry may use another approved provider if tenant policy allows it. Record the provider, model version, settings, input hash, retrieved context, and output. We promise a replayable evidence trail, not identical wording from a statistical model.
Logical run state machine
Attempt failure does not equal logical failureAmbiguous side-effect sequence
At-least-once transport, effect-aware workflowLease + fencing
A monotonically increasing lease epoch prevents a zombie worker from committing after failover.
Checkpoint contract
State is hash-addressed, encrypted, schema-versioned, and references immutable context—not raw mutable session memory.
Cost ledger
Reserve before scheduling; append usage by provider receipt; finalize or release reservation. Dashboard may lag the ledger.
0824:00–28:30
Failures, security, and recovery
A strong candidate changes the design when the constraint changes. Use the scenario console to rehearse.
Interviewer
GitHub sends a push three times, then an older event. Mid-index, the installation is revoked. A force-push deletes commits already referenced by audit edges.
Candidate
Candidate · clear top-down answerFor a code-host webhook, verify the signature, save its delivery ID, and ask the host for the real current commit. Do not trust event order. When access is removed, block queries immediately and prevent old workers from publishing anything new.
Interviewer
An EU bank wants a dedicated VPC, tenant-managed keys, no raw code in central logs, zero-retention models, and complete derived-data deletion. What changes?
Candidate
Candidate · clear top-down answerKeep EU customer content and processing inside the EU. Give each tenant separate encryption keys. Allow only approved model and tool destinations. For deletion, follow every derived copy—search chunks, embeddings, checkpoints, and test data—then remove keys and produce a completion receipt.
Interviewer
One region fails with 2,000 jobs active. A customer then submits 20,000 migration jobs. Prevent split brain and starvation.
Candidate
Candidate · clear top-down answerIf one region fails, only the current scheduler may hand out work. A new region takes control through shared agreement and gives workers a new lease number, so old workers cannot commit. Reserve capacity for interactive jobs and cap large migrations per tenant.
Nominal run
Pinned context + valid lease + available budget are required before execution.
Primary risk
Nondeterministic output is accepted without evaluation or trace evidence.
Architecture revision
Model output becomes a suggestion; tests and human approval gate merge.
Trust boundaries
Human edge
- SSO / MFA
- device and session
- org/project role
- intent + approval
Control plane
- policy decision
- budget + leases
- immutable manifest
- audit identity
Retrieval
- tenant ACL
- classification
- source lineage
- prompt-injection labels
Sandbox
- ephemeral worker
- no ambient creds
- egress allowlist
- resource limits
External
- scoped OAuth
- effect idempotency
- provider retention
- receipt + reconcile
| Threat / failure | Prevention | Detection | Recovery |
|---|---|---|---|
| Cross-tenant retrieval | Tenant in partition key + policy-filtered query + tenant key | Canary documents, negative auth tests, access-anomaly alerts | Revoke grant, isolate index, incident audit, rebuild |
| Prompt injection in code | Retrieved text typed as data; tool capabilities independent of text | Policy-denied tool-call metrics; suspicious instruction classifier | Stop run, preserve trace, update skill/policy, re-evaluate corpus |
| Zombie worker after failover | Lease/shard epoch fencing at every commit and privileged tool call | Stale-epoch rejection count | Requeue from checkpoint; investigate duplicate attempts |
| Search ACL lag | Query-time and post-fetch authorization, not index filter alone | Revocation propagation SLO | Deny on uncertain policy; rebuild affected shards |
| Audit tampering | Append-only storage, integrity hash checkpoints, segregated writer role | Continuous chain verification | Fail closed for release export; restore replicated ledger |
| Deletion residue | Source-lineage IDs and tenant key generation on every derivative | Deletion coverage query + sampled forensic scan | Crypto-erase, rebuild, attestation, incident if SLO breached |
Classic traps to name before they name you
“Use a graph DB.”
A product choice without typed edge semantics, version identity, traversal budget, or authorization is not a design.
“Exactly once.”
External business effects need idempotency receipts, reconciliation, and sometimes a human—not a messaging slogan.
“Temperature zero.”
Provider/version changes, hidden infrastructure, and nondeterminism prevent byte-identical replay. Preserve observable evidence.
“CRDT everything.”
Text convergence is not semantic approval. Publish, money, leases, and permissions need transactional authority.
“Cache permissions.”
Revocation must beat stale cache. Deny when policy freshness is unknown; authorize after retrieval too.
“Retry the agent.”
Without attempt identity, checkpoint, effect ledger, cancellation fence, and budget reconciliation, retry creates harm.
0928:30–30:00
Evaluation, rollout, recap
Close by proving the system changes outcomes safely, not by listing dashboards.
Interviewer
How do you know this system is useful and safe? Give me your rollout and final summary.
Candidate
Candidate · clear top-down answerI measure API health, search delay, queue age, worker recovery, cancellation time, link coverage, drift accuracy, reviewer load, cost, defects, and rollbacks. I would launch read-only indexing, then advisory warnings, then sandboxed agents, and only later allow policy-gated automation.
Control plane health Example scorecard
API SLO
99.95
P95 queue age
42s
Checkpoint recovery
91%
Missing receipts
0.03
Product quality Measure accepted outcomes
Trace coverage
94%
Drift precision
83%
Review burden
36m
Accepted cost
$7.60
Illustrative values only. A useful metric always defines unit, window, slice, owner, and action threshold.
Observe
Read-only repo indexing. Validate ACL, freshness, cost, and parser coverage. No workflow effects.
Advise
Provenance suggestions and drift findings. Capture dismissal reasons and calibrate thresholds by slice.
Execute safely
Agents run tests and open PRs in sandboxes. Human merge required. Chaos-test leases and receipts.
Gate by policy
Enable selected low-risk automations per tenant, with budgets, approvals, kill switches, and rollback.
Your 10-point closing checklist
- Define authoritative truth and human boundaries.
- Separate stable identity from immutable version.
- Pin releases and runs to exact manifests.
- Commit outbox work with truth.
- Let derived indexes converge visibly.
- Model logical runs separately from attempts.
- Fence leases, cancellation, and side effects.
- Authorize retrieval and tool use independently.
- Measure outcome quality and tenant fairness.
- Roll out from observe to advise to gated action.
88design signal
when all invariants land
when all invariants land
One-minute rehearsal
Say the final candidate paragraph aloud. Then name the three principal trade-offs: eventual impact/search, no global exactly-once, and audit replay based on observable evidence rather than identical model text.