One Product. Four Sovereignties.

I’ll separate a global product/control plane from independently survivable regional or customer data planes. “Same product” means one signed release contract and conformance suite, not identical infrastructure. I need the fleet mix, regulatory boundaries, customer approval windows, maximum version skew, telemetry permitted per mode, data classifications, RPO/RTO tiers, and whether 8090 or the customer owns incident actions.

My invariant: no deployment fetches mutable “latest” artifacts. Desired state pins content hashes for code, schemas, config, policy, and models. Data planes continue serving safely if the control plane is unavailable; air-gapped sites never require runtime connectivity.

Actors: product/release engineers, tenant admins, security/compliance, site reliability, customer VPC/site operators, support, auditors, and automated reconcilers. Required: tenant/deployment inventory; placement and residency policy; signed hermetic bundles; separate config/policy schemas; approval and rollout rings; preflight, backup, migration, health gate, rollback; drift detection; tenant key references; metrics/log/trace contracts; incident and audit export; DR plans and restore evidence; offline import/export.

Mock targets: 1,200 deployments across 12 regions; shared tier 99.95%, dedicated critical tier 99.99%; control-plane loss must not break serving; shared RPO <5 min/RTO <30 min, dedicated tiered, air-gap customer-agreed; critical CVE patch to connected fleet in 24 hours and offline bundle in 72. Out of scope: one-off forks per customer, silent breaking config changes, remote shell into air gaps, and universal raw-log export.

A no-pause fleet pass is 30 waves and 600 minutes—10 hours. Two percent means about 24 interventions, enough to overwhelm a small on-call if we do one global blast; canary rings and automatic halt are mandatory. Telemetry is 36,000 samples/minute, 600/s, about 600 KB/s or 51.8 GB/day before replication and index amplification. Logs/traces can be orders of magnitude larger, so budgets, sampling, and tenant-safe labels matter.

The fleet unit may be a shared regional cell rather than one tenant. I size and roll out by fault domain and risk tier, not only count. Air-gap upgrades have human lead time, so supported-version windows and compatibility matrices are a product feature.

A ReleaseBundle is immutable: application images, migration steps, model/assets, SBOM, provenance attestation, compatibility ranges, conformance results, and signature. A ConfigSet contains nonsecret operational values validated by a schema. A PolicySet is effective-dated business/security behavior with its own approvers. A SecretRef points to a deployment-local secret manager or customer KMS; secret bytes never enter global desired state.

DeploymentDesiredState pins all four digests plus placement, release channel, maintenance window, and data/key policy. Reconciler reports observed state and health with a monotonic generation. Rollout attempts and approvals are append-only. APIs create signed promotions, propose desired-state changes with expected generation, approve, pause, rollback, report drift, and export offline bundles/attestations.

The global control plane holds product catalog, deployment inventory, release metadata, compatibility, rollout policy, and redacted fleet health. Artifacts are built hermetically, scanned, signed in a protected release service, and replicated to regional registries or exported as a signed offline package. Regional/data-plane reconcilers pull desired state, verify signatures and policy locally, take a backup/checkpoint, run preflight, deploy dark, migrate with expand/contract schemas, health-check, then atomically switch traffic.

Shared SaaS uses multi-tenant regional cells; dedicated uses one isolated stack/account; VPC uses a customer-side agent with outbound-only control channel or customer-triggered pull; air gap uses an import station and local reconciler. Observed state returns asynchronously where permitted. Control loss leaves the last approved state serving; local safety controllers can rollback without global permission and later upload receipts.

Rollout rings are internal test, one synthetic cell, employee tenants, low-risk shared cells, representative dedicated/VPC canaries, broad connected fleet, then offline channels. Advancement requires a minimum soak, SLO burn, migration reconciliation, and domain quality gates; failures halt the ring automatically. We cap concurrent units per region/account/customer and never consume all redundancy at once.

Database changes use expand/contract: add backward-compatible structures; deploy code that can read old/new and dual-write if necessary; backfill with throttling and checkpoints; verify counts/invariants; switch reads; only remove old schema after the minimum supported version ages out. Rollback changes the application route and feature flag; irreversible data transformation requires preimage/archive and a forward-fix plan. Bundle compatibility says which code can read each schema/config/policy version.

The customer-side reconciler has a narrow, documented contract: verify approved bundle, apply desired state, collect redacted health, and produce signed receipts. It cannot access application data by default. VPC connectivity is outbound mTLS with customer allowlists, short-lived workload identity, and optional private endpoints. Support uses explicit time-bound customer approval, command allowlists, session recording, and break-glass—not persistent vendor SSH.

An air-gap bundle contains images, dependencies, models, migrations, SBOM, vulnerability data, schemas, conformance fixtures, and signatures. The import station validates chain of trust, anti-rollback policy, completeness, hardware/runtime compatibility, and clean-room tests. Telemetry export is a customer-reviewed, schema-limited signed package with aggregates and redaction; diagnostics can be generated locally. We publish offline security advisories and delta/full bundles, and receive an installation/health attestation when policy permits.

The health gate was inadequate. I halt the rollout automatically on the domain invariant signal, isolate affected units, and keep unaffected units on the prior desired generation. For each affected deployment, the runbook inspects migration journal and index reconciliation. If the old code remains schema-compatible, route back and rebuild the derived index from authoritative data; if not, execute the tested forward-fix under incident approval. We never rerun a non-idempotent migration blindly.

Future gates include synthetic business transactions, count/checksum invariants, query-result parity, background job health, and a representative soak. Each migration step has a unique ID, precondition, checkpoint, postcondition, retry class, and compensating/forward action. The 2% pattern is sliced by database/version/region to find a compatibility cohort before resume.

Serving remains on last desired state. Release signing and artifact distribution have an emergency continuity path separated from general orchestration, with dual control and pre-authorized procedures—not an unsigned shortcut. Regional reconcilers can accept a signed emergency bundle from replicated catalogs. VPC customers receive impact, compatibility, and compensating-control guidance; we respect their authority and record risk acceptance rather than forcing a backdoor.

For air gaps we build patches against every supported base version or require an explicit staged upgrade, then ship signed full/delta bundles through the agreed channel. Sites validate locally and return attestation later. If an old version is outside support, that is visible in the fleet risk ledger before the incident. The long-term fix is support-window enforcement, dependency inventory/SBOM search, and regular offline upgrade rehearsals.

Placement policy is evaluated before provisioning and continuously: tenant data, backups, indexes, logs, and keys remain in allowed regions. Tenant envelope keys wrap data keys; dedicated/VPC may use customer-managed KMS grants. Key rotation is versioned and resumable. Cryptographic erasure is valid only when the encryption-key scope exactly matches the deletion target; a key shared by lawful retained records cannot be destroyed, so those records require object/row deletion plus backup-expiry and tombstone reconciliation under retention/legal-hold policy. Queue and telemetry labels use deployment IDs, never raw tenant data.

Reconcile uses generation and operation IDs; completed steps are journaled so retries are idempotent. Rollout queues enforce per-fault-domain concurrency, maintenance windows, retry budgets, and emergency priority without starving repair. Observe desired/observed drift, version skew, upgrade stage age/failure, SLO burn by cell, migration invariants, telemetry gaps, backup freshness/restore success, key errors, residency violations, and support access. DR is per tier: cell evacuation for shared, warm region or documented rebuild for dedicated, customer-coordinated VPC/air-gap; quarterly restore proves RPO/RTO.

First standardize release manifests and conformance without changing deployment; onboard shared internal cells; add dedicated instances; then VPC agents with two customers; finally offline bundle/import/receipt workflows. Run fleet inventory and drift in read-only mode before automatic reconciliation. Prove backup restore and rollback for each mode. Publish support/version policy and error budgets jointly with customers.

The trade-off is fleet uniformity versus customer sovereignty. Forcing one version simplifies operations but violates change control and air-gap reality; unlimited skew multiplies vulnerability and test matrices. I set explicit supported windows, compatible expand/contract contracts, strong incentives and visibility, while preserving customer approval and a narrow emergency process.

I’ll summarize immutable desired state, independent data planes, ringed upgrades, sovereignty, and recovery.