Realtime Collaborative SDLC

I will separate two user promises. Drafting should feel local-first and converge under concurrent human and agent edits. Publishing should create one immutable, schema-valid revision that downstream systems can trust. So I use a CRDT-like operation model for drafts, server validation plus compare-and-swap for publication, and explicit suggestions for agent work. Before choosing algorithms I would ask whether arbitrary rich text or structured domain blocks dominate.

The first release supports typed requirement, acceptance-criterion, component, API and decision blocks; reorder and nesting; inline text; comments; suggestions; presence; offline edits; immutable publication and revision diff. An accepted suggestion is a new operation batch, not a hidden overwrite. Agents cannot publish. Cross-document references may be temporarily unresolved in a draft but must satisfy policy at publish. Presence is ephemeral. I exclude source-code editing, video, public sharing and autonomous approval.

• Draft guarantee: eventually convergent, with visible conflicts when intent cannot be merged safely.
• Publish guarantee: one schema-valid immutable content hash.
• Downstream consumers pin revision numbers, never “latest draft”.

The mock peak is 8,000 operations/s. If a compressed operation averages 350 bytes including metadata, raw ingress is 2.8 MB/s or about 242 GB/day before replication. Keeping every granular operation for seven years would be wasteful, so retain 90 days of ops, compact snapshots, and preserve approved revisions permanently. At 1M documents, an average 40 blocks gives 40M blocks. Hot-room fan-out is the harder problem: 150 participants receiving 20 ops/s means 3,000 deliveries/s for one active document.

8,000 ops/s × 350 B ≈ 2.8 MB/s
Per day ≈ 242 GB raw
Hot room: 20 ops/s × 150 peers = 3,000 deliveries/s

Document holds identity, type and current draft epoch. Blocks have stable UUIDs, kind, parent and a fractional rank so moves do not renumber siblings. Operations are append-only and deduped by actor/clientOpId within an epoch. Snapshots record state plus causal clock. Suggestions preserve their base clock and patch. Comments anchor to block ID plus a relative text position and context hash. PublishedRevision stores immutable canonical content, source clock, approvers and content hash. ReferenceEdge is generated from the published revision for impact analysis.

• Deleting a block creates a tombstone until compaction, preserving concurrent references.
• Human identity and agent identity are distinct actors with delegation metadata.
• Published revision is content-addressed and never edited in place.

A CRDT gives offline availability and convergence without a single low-latency primary for every keystroke. I would use a sequence CRDT inside rich-text fields and stable block IDs with conflict-free order for the tree. But convergence is not semantic validity: two edits can each be valid and jointly violate “unique requirement ID” or create a reference cycle. Therefore draft replicas converge first, a deterministic validator annotates invariant conflicts, and publish rejects unresolved blocking issues. Approval, role changes, suggestion decisions and publication stay serialized transactions.

The client applies an operation locally, assigns actor sequence and causal context, persists it in an offline queue, then sends to the nearest realtime edge. The edge authenticates tenant/document permission and ACL epoch, dedupes clientOpId, and appends to a document-partitioned durable log. Room fan-out sends the accepted op immediately. Materializers consume in document order, build snapshots, update search and run incremental schema checks. Presence uses an expiring in-memory channel and is not logged as truth. Publish goes to the owning document transaction service, verifies expected draft clock and approvals, canonicalizes content, validates cross-document policy, writes PublishedRevision and an outbox event atomically.

• Partition durable operations by document ID.
• Route a hot document across fan-out nodes while keeping one logical operation order.
• Derived notifications, search and reference graphs consume the outbox.

Each operation names a stable block and a typed intent: insert block, move block, set scalar, splice text, add reference, tombstone. Syntactic merge is deterministic. Semantic validators then evaluate unique IDs, allowed parents, required fields, contract compatibility and cycles. Nonblocking violations appear inline. Blocking violations prevent publication. For fields where last-writer-wins would erase intent—such as an API method or data classification—I use a multi-value conflict register that shows both proposals and requires resolution. Schema migrations are versioned transforms over snapshots, tested for determinism and reversible before rollout.

• Stable identity survives move and rename.
• Canonical serializer makes equal logical state hash identically.
• Validation result pins schema and policy bundle versions.

A comment anchor includes block ID, CRDT-relative start/end positions and a small context hash. Text operations transform the relative positions. If the block is deleted, the thread becomes orphaned but remains visible in document history with surrounding context. A suggestion is not a private fork of the whole document; it is a typed operation patch against a base clock. The server previews the rebased patch. Accept performs a compare-and-swap on suggestion decision version, translates the patch onto current state, reruns validation, and appends the resulting operation batch attributed to the reviewer and original author.

• Two reviewers cannot both accept the same suggestion.
• Partially conflicting suggestions require an explicit revised patch.
• Mentions are derived notifications, not authorization grants.

Draft operation delivery is at least once and eventual; actor sequence plus clientOpId makes replays harmless. The append log gives an accepted operation durable order per document, while replicas may display causally ready ops early. Strong transactions protect ACL changes, publish, approval, suggestion decision and billing. Publish includes expected draft vector/hash and idempotency key; a duplicate returns the same revision. If new operations landed, it returns 409 with the new clock rather than silently publishing a different document.

The agent returns a patch based on an old clock, never raw “replace document” content. Stable block IDs let the rebaser apply independent edits. Operations against moved blocks follow identity; edits against deleted or type-changed blocks become conflicts. The system produces a preview classified as clean, auto-rebased with warnings, or needs-human-resolution. It does not let a stale agent publish or overwrite. For very old epochs after compaction or schema migration, the agent must retrieve the current snapshot and regenerate its proposal, while its original response remains auditable.

Realtime edges maintain room membership, but durable ops stay in the partitioned log. Fan-out uses coalesced operation frames and bounded per-client buffers; a slow client receives a catch-up cursor rather than unbounded memory. During reconnect, clients send last clock and pending client IDs. The server returns a compact snapshot plus only missing ops. Add jitter, admission control and a per-tenant reconnect budget. Presence degrades first. Editing remains locally available; publication may be briefly unavailable if the owning transaction shard is recovering.

• Snapshot every N ops or bytes, tuned by replay budget.
• Hot-doc detection adds fan-out replicas but does not split document truth.
• Backpressure tells clients to batch cursor/presence noise.

Approval is bound to a content hash and policy bundle, not a floating document. The agent edit advances the draft clock and invalidates approvals whose policy says the changed block matters. The first publish transaction checks expected content hash and approval set; it either publishes exactly that reviewed state or returns stale. The second publish with the same idempotency key returns the existing revision; a different key receives a conflict. The UI shows “approval expired by blocks X and Y” so users understand why.

Authorization is enforced on session join, every reconnect, publish, export and notification. ACL epochs fence long-lived sockets; permission changes force re-auth. Document and operation keys include tenant ID, and storage/search partitions carry tenant labels. Agent tokens are scoped to tenant, project, document, action and expiry; delegated identity records the human or service that invoked them. Encrypt approved revisions with tenant keys, sanitize rich content, scan uploads, and treat agent-authored links/instructions as untrusted. Audit reads of sensitive documents, approval, export and permission changes.

• A WebSocket is not permanently authorized because it was valid at connect time.
• Mention does not grant access.
• Cross-document reference resolution cannot reveal a forbidden title.

Measure local-to-ack and accepted-to-peer latency, op-log lag, reconnect duration, snapshot replay bytes, dedupe rate, validation latency, publish conflicts, lost-anchor rate and hot-room buffer drops. Correctness uses deterministic convergence tests: shuffle, duplicate and delay operations across replicas and assert identical canonical hash. Fuzz tree moves, deletes and schema migrations. Product quality includes suggestion acceptance, manual-conflict rate, time from draft to approval and approval invalidation frequency. Audit jobs verify every published hash can be reconstructed.

• SLO: no acknowledged operation lost.
• Invariant dashboard: duplicate requirement IDs and dangling published references target zero.
• Canary schema migrations on cloned snapshots.

Ship single-user structured editing and immutable versions first, then multi-user online collaboration, then offline, comments, suggestions and agents. Run a shadow canonicalizer before trusting hash reconstruction. CRDT complexity is justified by offline and concurrency needs; if those disappear, a simpler server-ordered OT design may win. Traps are using last-write-wins for meaningful fields, treating presence as durable, anchoring comments only by character offsets, allowing agents to replace documents, and binding approvals to “latest.”

LivingSpec keeps a highly available convergent draft and a strongly controlled published truth. Clients optimistically apply typed CRDT operations; a document-partitioned log durably orders accepted ops, materializers snapshot and validate them, and ephemeral presence stays outside truth. Stable block IDs preserve moves, comments use relative anchors, and agents submit reviewable patches against base clocks. Publication validates schema, references, approvals and expected content in one transaction, producing an immutable hash. Epoch fencing, dedupe, convergence fuzzing and gradual rollout make human-plus-agent collaboration trustworthy.