Keep agents moving. Fence every effect.

I will separate the workflow authority from disposable execution. A logical run pins tenant/project, workflow version, context manifest, capability profile, budget, priority, and cancellation epoch. Each ready step may have attempts; a scheduler issues a fenced lease. Workers heartbeat, checkpoint, and request model/tool operations through gateways that validate the current run/lease/cancellation state.

I need to clarify DAG expressiveness, dynamic fan-out, maximum recursion, step determinism, checkpoint contract, approval semantics, external-effect risk, ordering, deadlines, tenant priority classes, model/tool quotas, regional/residency constraints, and whether a provider change is allowed. I will optimize useful accepted outcomes—not worker utilization alone.

Actors are user, project admin, automation owner, agent/subagent, reviewer, scheduler operator, model/tool provider, and auditor. Functional flows: define/version workflow DAG; launch idempotently; compute readiness; reserve quota/budget; run/pause/resume/cancel; spawn bounded subagents; checkpoint; wait for event/time/human approval; stream progress; retry infrastructure failures; reconcile external effects; collect artifacts; and export run lineage.

NFRs: P95 interactive start under 10 seconds when capacity exists, cancellation visible under two seconds and enforced at the next effect boundary, 99.95% control-plane availability, no lost accepted run, 30-second failover detection, tenant fairness, region/residency, and seven-year audit for consequential effects. The scheduler does not implement foundation models, Git providers, or test runners; they are capability-gated adapters.

1. Retrying launch returns one logical run and one initial budget reservation.
2. Only the current run/step state version and lease epoch may commit progress, checkpoint, or invoke a privileged effect.
3. DAG readiness requires every configured predecessor condition; cycles and unbounded dynamic expansion are rejected.
4. A checkpoint names exact workflow/context/tool/model artifact versions and the last completed logical boundary.
5. Cancellation increments a monotonic epoch checked before model/tool calls and checkpoint commit; terminal cancellation cannot be resurrected.
6. External effects are not “exactly once” by messaging claim: intent precedes call, receipt follows, unknown outcomes reconcile.
7. Subagents inherit an intersection of parent capabilities and a bounded slice of parent budget/depth/fan-out.
8. One tenant’s migration flood cannot consume every interactive slot or provider quota.

Ten thousand active attempts heartbeating every 20 seconds produce 500 heartbeats/s. At 512 KB checkpoints every two minutes, peak ingress is ~42.7 MB/s decimal (about 41.7 MiB/s) and 3.69 TB/day if retained uncompressed—so we checkpoint at logical boundaries, delta/compress, keep latest plus policy milestones, and compact successful runs.

One hundred thousand runs × 20 steps is two million step executions/day, only 23 starts/s average, but fan-out and tenant bursts dominate. If average run duration is 45 minutes, Little’s Law gives ~3,125 average concurrent runs; 10,000 peak is plausible. Queue partitions, provider quotas, long-held approvals, and checkpoint storage—not raw CRUD QPS—drive the design.

WorkflowVersion contains a validated DAG and step contracts. Run owns logical intent and monotonic state/cancellation versions. StepRun owns readiness, dependencies, policy, and accepted result. Attempt owns worker, lease epoch/expiry, heartbeat, and failure. Checkpoint owns content digest and resume cursor. EffectIntent, ApprovalGate, BudgetReservation, and SubagentLink keep hard boundaries explicit.

Every command uses idempotency plus expected state version. Claim uses atomic compare-and-set from READY to LEASED with a new epoch. Heartbeat/commit requires run, step, attempt, lease epoch, and cancellation epoch. A transaction changes authoritative state and writes an outbox event; UI streams and search are projections.

Launch validates workflow/context, reserves parent budget and tenant concurrency, stores Run + initial ready steps + outbox, then returns. The readiness service consumes terminal step events and transactionally enqueues newly satisfied steps. Hierarchical queues classify interactive, automation, migration, and approval-resume work. A scheduler selects by tenant fair share and capability/resource needs, then leases an attempt.

The isolated worker restores a signed checkpoint and fetches only manifest-authorized context. Model/tool gateways enforce run/lease/cancel epochs, data policy, budget, and capability. Spawning a subagent is an authoritative command that checks depth/fan-out and carves a child budget/capability intersection; the parent waits on an explicit join policy. Before a PR/deploy/tool write, the workflow creates an approval task. The reviewer’s signed decision wakes the step or cancels it. Progress streams can fail while durable work continues.

Readiness is a deterministic projection from completed predecessor conditions, workflow version, approvals, cancellation, and artifacts. It produces a ReadyTicket containing tenant, project, class, deadline, resource/capability needs, estimated cost, and enqueue time. It does not reserve a worker indefinitely while waiting for provider quota.

Scheduling is hierarchical: reserved interactive capacity, then tenant weighted fair share, then project/user quotas, priority/deadline, and aging. Migration has a concurrency cap and can use otherwise idle capacity but is preemptible only at checkpoints—not arbitrary process kill. Separate token buckets protect model/provider/tool rate limits. Dominant-resource fairness can account for CPU, memory, model tokens, scarce licensed tools, and regional pools.

Admission controls dynamic DAG expansion: max child count, depth, total steps, estimated token/tool budget, and tenant active-descendant quota. The parent cannot recursively manufacture priority. Metrics include queue age and slowdown by tenant/class, starvation count, and useful completion—not only fleet utilization.

Before the tool call, the gateway transactionally records EffectIntent(effect_key=run/step/pr/repo/branch, REQUESTED) under the current lease/cancel epochs. It passes the key/correlation to the SCM connector. The provider creates PR 184, but response is lost. The worker dies; its lease expires.

A new attempt restores the last checkpoint and sees a REQUESTED effect without receipt. It queries the provider by idempotency/correlation/branch-head. If PR 184 exists with matching request hash, it records OBSERVED and continues. If no lookup exists, the step enters NEEDS_RECONCILIATION for a human rather than opening another PR. A late zombie worker cannot commit because its lease epoch is stale.

Checkpoint state contains completed logical step cursor, artifact digests, conversation/tool summaries needed to resume, pending effect IDs, and provider/model configuration—not raw process memory or hidden model chain-of-thought. Provider failover creates a new attempt/config record and may require re-evaluation or approval.

The cancel command transactionally increments run.cancellation_epoch, changes logical state to CANCEL_REQUESTED, appends outbox, and returns. The UI is explicit: requested versus enforced. Workers subscribe, but correctness does not depend on delivery. Every privileged gateway call and checkpoint commit includes the epoch and is rejected if stale; long operations receive cancellation tokens where providers support them.

If the deployment request was recorded before cancel, effect policy decides whether to let it finish, issue compensation/rollback, or pause for reconciliation. Cancellation cannot erase an external effect. The run reaches CANCELED only when active leases are fenced/expired and all effect intents are terminal or explicitly reconciled. Child runs inherit cancellation unless marked independent by approved workflow policy.

Each scheduler shard has one consensus-backed authority epoch. Failover requires quorum and increments the shard epoch. New leases contain the new shard+lease epoch; model/tool gateways and checkpoint commits reject old-region epochs even if workers keep running. We prefer a write pause over dual authority.

Run/step/effect records replicate with an RPO appropriate to side effects—effect intent must be durable before external call. Checkpoint objects are cross-zone and, where residency allows, asynchronously cross-region. The standby waits for old leases to expire or explicitly fences them, rebuilds ready/leased projections from authoritative state, reconciles in-flight effects, and requeues from latest valid checkpoints. Progress streams and noncritical analytics may be unavailable.

Recovery metrics include fenced-zombie rejections, duplicate attempts, checkpoint loss window, unknown effects, time to useful completion, and tenant fairness after backlog—not merely control-plane HTTP uptime.

Workers are ephemeral, tenant/project/run scoped, without ambient cloud credentials. Context retrieval and every model/tool call pass through policy; retrieved code/doc text is untrusted data, never capability. Secrets are exchanged just-in-time for exact tool/action/resource and are not checkpointed. Sandboxes enforce filesystem, network egress, process, CPU, memory, disk, and time limits.

Subagent creation intersects parent capabilities, restricts source context, allocates child budget, and increments depth/fan-out counters transactionally. A child cannot choose its own tenant/project or promote priority. Tenant IDs and data classes propagate through queue records, object keys, logs, metrics, model caches, and artifacts. Audit records observable inputs/config/tool calls/outputs/approvals—never claims hidden reasoning as reliable evidence.

Health: launch/readiness latency, queue age/slowdown by tenant/class, lease acquisition/expiry, heartbeat load, checkpoint latency/bytes/recovery, zombie rejection, cancellation enforcement, unknown effect rate, provider/tool saturation, budget variance, recursion rejection, and regional failover RTO/RPO. Quality: accepted artifact rate, test/approval pass, retries per accepted outcome, cost and wall time, human reconciliation burden, escaped defects, and fairness.

Rollout begins with short idempotent read-only steps, then checkpoint/retry without external effects, then safe idempotent tools, then approved consequential writes, then multi-region failover. Fault injection kills workers, drops heartbeat/response messages, delays queues, corrupts checkpoints, duplicates events, exhausts providers, races cancel, and partitions scheduler authority. Every phase has per-tenant kill switch and drain mode.

Runway Control stores one durable logical run with pinned workflow/context/policy/budget and many fenced step attempts. DAG readiness creates tickets; hierarchical multi-resource fair queues choose work. Workers restore versioned checkpoints, while model/tool gateways enforce current lease and cancellation epochs. Subagents inherit narrower capabilities and bounded budget.

External effects use intent, idempotency/correlation, receipt, and reconciliation; no exactly-once fiction. Regional failover increments authority epochs so zombies cannot act. The trade-off is checkpoint frequency versus cost/recovery: checkpoint at meaningful boundaries with compaction, and accept bounded rework while never replaying an unknown harmful effect blindly.