Design an auditable AI software factory.

I will first define what Forge is authoritative for. I hear three systems with different failure semantics: a collaborative system of record, a rebuildable code/search intelligence plane, and a long-running workflow engine that touches unreliable external systems.

Before drawing boxes: are published artifacts immutable? Is drift advisory or release-blocking? What may an agent do without human approval? Must audit replay reproduce an identical model output, or prove the exact observable inputs, policy, calls, outputs, and approvals?

Good. I will optimize the control plane for correctness and availability, while keeping expensive fan-out asynchronous. My north-star invariant is: every material output names the immutable input-version manifest and the principal, policy, model/tool configuration, and approvals that produced it.

I will keep model training and language-parser implementation out of scope, but define provider and indexing interfaces. I will also treat repository text as untrusted data; retrieved code cannot grant itself more tool permissions.

My critical write path is author, review, and publish immutable artifact versions; create and transition work orders; reserve budget; acquire a run lease; and record approval or external side-effect receipts. These require transactional checks.

Search, semantic embeddings, symbol indexing, drift, analytics, notifications, and large impact closure are derived. They may converge, but must expose generation and freshness. The user should never confuse “index caught up” with “truth committed.”

Yes. The publication transaction writes the immutable version, appends an outbox event, and returns. An impact worker consumes the event, traverses a bounded snapshot, stores findings by analysis generation, and emits progress. A release gate can require generation G to complete before release, without holding publication hostage.

At 1,000 normal and 5,000 peak authoritative mutations per second, the primary store is not extraordinary, but each logical action can emit audit, outbox, notification, graph, and index work. At a conservative 8 derived events per mutation, peak internal ingress is about 40,000 events per second.

Fifteen thousand webhook deliveries per second at roughly 2 KB each is 30 MB/s before repository fetches. I acknowledge after signature verification and durable dedupe, then coalesce by repository/ref and reconcile to the provider's current commit rather than processing arrival order.

Five thousand active runs with a 30-second heartbeat is about 167 heartbeats/second. If each produces a 256 KB checkpoint every five minutes, checkpoint ingress is only about 4.3 MB/s—but retaining every checkpoint indefinitely would be wasteful, so I compact successful runs and preserve policy-relevant receipts.

Artifact is a stable logical object. ArtifactVersion is immutable content and structured metadata addressed by version ID and content hash. A Publication pins a reviewed set of version IDs. “Latest” is a UI projection, never a release dependency.

A ProvenanceEdge connects specific version/span endpoints and records type, asserter, evidence, confidence, and validity. Work-order dependencies are separate because they have workflow invariants such as acyclicity and readiness.

For agents, AgentRun is the logical intent; RunAttempt represents a worker try. Checkpoints, model calls, tool calls, side-effect receipts, budget reservations, approvals, and audit events all key back to tenant, project, run, and attempt.

The client edits through a realtime collaboration service, but publication goes to the authoritative API with an expected base version. In one transaction it stores the immutable version, advances the current projection, and writes an outbox record.

The event backbone fans out to impact analysis, search indexing, notifications, and policy evaluation. A human creates or updates a work order pinned to exact requirement and blueprint versions. The scheduler reserves quota and budget, snapshots an authorized context manifest, and leases an attempt to an isolated worker.

The worker retrieves only manifest-approved data through a policy gateway, invokes a model through a provider-neutral gateway, and calls tools with scoped capabilities. Tests and a PR are evidence artifacts; neither becomes approved truth until a human/release policy records a decision.

Publication of v8 is immediate and does not mutate v7. An impact job starts from edges incident to v7 and its stable element IDs, follows only policy-relevant edge types with cycle detection, and writes findings against analysis generation ag_92.

Running work retains its pinned v7 context manifest. Policy evaluates the changed field: a safety constraint is high severity, so affected runs move to STALE_PENDING_REVIEW before their next privileged tool step. Open PRs receive a visible stale-context check. A human may explicitly accept old-context output, but that exception is an approval event and the release still states it satisfies v7 unless new tests prove v8.

We incrementally compare stable requirement element IDs, not whole documents, to avoid invalidating unrelated work. Release gates query trace coverage: every v8 acceptance criterion needs a passing test and approved implementation edge at the pinned commit.

A graph engine may help traversal, but it does not define authority, version semantics, access policy, or a bounded query. I store the canonical edge fact with transactional identity and publish it through an outbox. A graph projection can be optimized and rebuilt. Closure is bounded by tenant, release snapshot, edge types, depth, node budget, and cursor; huge traversals return partial progress instead of blocking publication.

The laptop owns only a subscription to progress. The durable run record, encrypted context manifest, budget reservation, checkpoints, and side-effect ledger live in the control plane. A scheduler grants an attempt a time-bounded lease with an epoch. The worker heartbeats and checkpoints at logical boundaries.

On missed heartbeats, the scheduler expires the lease and starts a new attempt from the latest valid checkpoint. Fencing means the old attempt cannot commit later. Cancellation increments a cancellation epoch; every tool gateway and checkpoint commit rejects stale epochs. Progress streaming may be down while execution continues.

I cannot promise exactly-once across a system we do not control. Before the call, the tool gateway commits a REQUESTED receipt with a stable effect key such as run/step/tool/logical-target. It passes that key to providers supporting idempotency. After success it records the external ID and response hash.

If the outcome is unknown, the retry first queries by idempotency key or external correlation. If the provider offers neither, the run enters NEEDS_RECONCILIATION; a human verifies before retry. Compensation is a domain action, not a database rollback. We optimize for at-most-one harmful effect, not pretending transport semantics solve the business effect.

If tenant policy allows, a retry can route to a compatible provider after a capability and data-residency check. I record the provider, model/version, sampling parameters, normalized request hash, retrieved context IDs, and output. I claim functional evaluation and evidence—not byte-for-byte reproducibility. Privileged steps may require fresh approval after a provider change.

I verify the webhook signature against the installation secret, persist the delivery ID for dedupe, and enqueue by installation/repository/ref. A reconciler asks GitHub for the current ref, so delivery order is a hint—not truth. Parsing is keyed by immutable commit; duplicate work is harmless.

Revocation increments the integration grant epoch. Fetch and indexing workers re-check that epoch before every provider read and before publishing an index generation. New queries immediately fail policy checks even if a stale cache exists. Historical snapshots follow tenant retention policy: preserve encrypted audit references under legal hold or cryptographically erase content and leave a tombstoned hash/evidence envelope.

I deploy the data and execution planes inside the EU tenant boundary and keep a minimal global control record containing no customer content. Keys are envelope-encrypted by the tenant KMS; model/tool egress is allowlisted by policy. Authorization runs both before retrieval and before the model/tool call, preventing a confused deputy.

Every chunk, embedding, prompt cache entry, checkpoint, trace, and evaluation example carries source lineage and tenant/key generation. A deletion job traverses lineage, tombstones live references, destroys data keys, rebuilds indexes, and produces a completion attestation. Legal hold preserves event metadata, approvals, hashes, and signer identity—not deleted plaintext.

A single fenced scheduler authority per shard issues lease epochs from consensus-backed state. Failover requires quorum and increments the shard epoch; old-region workers cannot checkpoint or call privileged tools. The live event stream may disappear while safe runs continue.

Queues are hierarchical: tenant quota, workload class, then weighted fair scheduling with aging. I reserve capacity for interactive work, cap migration concurrency, and use per-tenant token buckets for model/tool budgets. I measure useful completion and queue age by tenant—not merely scheduler uptime.

I separate platform health from product quality. Health includes API SLO, index lag, queue age by tenant, lease recovery, cancellation latency, and missing receipts. Quality includes trace coverage, orphan rate, drift precision/recall, reviewer burden, requirement-to-test coverage, accepted work-order cost, escaped defects, and rollback rate.

I roll out in four gates: read-only repository indexing; advisory provenance and drift with measured reviewer feedback; durable agents limited to sandbox/test/PR; then policy-gated automation by tenant and action. Every gate has a kill switch, shadow comparison, and rollback.

In one sentence: Forge keeps immutable, tenant-scoped truth in a small transactional control plane; projects it asynchronously into search and impact views; and runs fallible agents through a durable, fenced, budgeted, evidence-producing workflow where humans retain authority.