One Product. Four Sovereignties.

I will split the system into two levels. A global coordinator publishes approved software versions and rollout plans. Each region or customer environment runs the product and keeps serving even if it cannot reach that coordinator.

No environment downloads a moving “latest” version. A release names the exact code, database changes, settings, policies, and models. Air-gapped sites install that same signed package without an internet connection. I also need to know who approves upgrades, how much downtime is allowed, and which health data may leave each environment.

Release engineers publish software. Customer and site operators approve and run it. Security teams set location and key rules. Reliability engineers and support teams monitor health and recover failures. Version one tracks every deployment, builds and signs exact release packages, rolls them out in small groups, checks health, detects drift, rolls back, and supports offline import and audit export.

For this mock, we manage 1,200 deployments in 12 regions. Losing the global coordinator must not stop customer applications. For the shared service, the recovery point objective (RPO) allows at most five minutes of lost work, and the recovery time objective (RTO) is under 30 minutes. A critical security patch reaches connected sites in 24 hours and offline sites in 72. We do not create permanent customer forks or hidden remote access to air-gapped sites.

A no-pause fleet pass is 30 waves and 600 minutes—10 hours. Two percent means about 24 interventions, enough to overwhelm a small on-call if we do one global blast. Canary rings and automatic halt are mandatory. Telemetry is 36,000 samples/minute, 600/s, about 600 KB/s or 51.8 GB/day before replication and index amplification. Logs/traces can be orders of magnitude larger, so budgets, sampling, and tenant-safe labels matter.

The fleet unit may be a shared regional cell rather than one customer. I size and roll out by fault domain and risk tier, not only count. Air-gap upgrades have human lead time, so supported-version windows and compatibility matrices are a product feature.

A ReleaseBundle is a package we never change after signing. It contains application images, database steps, models, a software bill of materials (SBOM), compatibility results, and the signature. A ConfigSet contains nonsecret operating values. A PolicySet contains dated business and security rules with separate approvers. A SecretRef points to a local secret manager or customer key service; secret values never enter the global coordinator.

DeploymentDesiredState names the exact release, configuration, policy, secret reference, location, maintenance window, and key rules wanted for one site. The local worker reports what is actually running with a sequence number that only increases. APIs promote, approve, pause, roll back, report drift, and export signed offline packages.

I would draw three parts: a global coordinator, a protected release builder, and a local deployment worker in each environment. The coordinator stores approved versions and rollout plans. The builder creates a complete package in a clean environment, scans it, signs it, and sends it to a regional registry or offline export. The local worker verifies the signature, checks local policy, takes a backup, installs without traffic, runs database steps and health checks, then switches traffic.

Shared software as a service (SaaS) uses regional cells. A dedicated customer gets an isolated stack. A virtual private cloud (VPC) uses a customer-side worker with an outbound-only connection. An air-gapped site imports the signed package manually. If the coordinator disappears, every site keeps serving its last approved version and may roll back locally.

Rollout rings are internal test, one synthetic cell, employee tenants, low-risk shared cells, representative dedicated/VPC canaries, broad connected fleet, then offline channels. Advancement requires a minimum soak, service target burn, migration cross-checking, and domain quality gates. Failures halt the ring automatically. We cap concurrent units per region/account/customer and never consume all redundancy at once.

Database changes use expand/contract: add backward-compatible structures. Deploy code that can read old/new and dual-write if necessary. Backfill with throttling and checkpoints. Verify counts/main rules. Switch reads. Only remove old schema after the minimum supported version ages out. Rollback changes the application route and feature flag. Irreversible data transformation requires preimage/archive and a forward-fix plan. Bundle compatibility says which code can read each schema/config/policy version.

The customer-side reconciler has a narrow, documented contract: verify approved bundle, apply target version and settings, collect redacted health, and produce signed receipts. It cannot access application data by default. VPC connectivity is outbound mTLS with customer allowlists, short-lived workload identity, and optional private endpoints. Support uses explicit time-bound customer approval, command allowlists, session recording, and break-glass—not persistent vendor SSH.

An air-gap bundle contains images, dependencies, models, migrations, SBOM, vulnerability data, schemas, conformance fixtures, and signatures. The import station validates chain of trust, anti-rollback policy, completeness, hardware/runtime compatibility, and clean-room tests. Telemetry export is a customer-reviewed, schema-limited signed package with aggregates and redaction. Diagnostics can be generated locally. We publish offline security advisories and delta/full bundles, and receive an installation/health attestation when policy permits.

The health gate was inadequate. I halt the rollout automatically on the key business-health signal, isolate affected units, and keep unaffected units on the prior desired generation. For each affected deployment, the runbook inspects migration journal and index cross-checking. If the old code remains schema-compatible, route back and rebuild the derived index from source of truth data. If not, execute the tested forward-fix under incident approval. We never rerun an unsafe-to-retry migration blindly.

Future gates include synthetic business transactions, count/checksum main rules, query-result parity, background job health, and a representative soak. Each migration step has a unique ID, precondition, checkpoint, postcondition, retry class, and compensating/forward action. The 2% pattern is sliced by database/version/region to find a compatibility cohort before resume.

Serving remains on last target version and settings. Release signing and saved item distribution have an emergency continuity path separated from general orchestration, with dual control and pre-authorized procedures—not an unsigned shortcut. Regional reconcilers can accept a signed emergency bundle from replicated catalogs. VPC customers receive impact, compatibility, and compensating-control guidance. We respect their authority and record risk acceptance rather than forcing a backdoor.

For air gaps we build patches against every supported base version or require an explicit staged upgrade, then ship signed full/delta bundles through the agreed channel. Sites validate locally and return attestation later. If an old version is outside support, that is visible in the fleet risk ledger before the incident. The long-term fix is support-window enforcement, dependency inventory/SBOM search, and regular offline upgrade rehearsals.

Placement policy is evaluated before provisioning and continuously: customer data, backups, indexes, logs, and keys remain in allowed regions. Customer separate encryption keys wrap data keys. Dedicated/VPC may use customer-managed KMS grants. Key rotation is versioned and resumable. Deletion revokes/wipes wrapped keys only under retention/legal-hold policy. Queue and telemetry labels use deployment IDs, never raw customer data.

Cross-check uses generation and operation IDs. Completed steps are journaled so retries are retry-safe. Rollout queues enforce per-fault-domain concurrency, maintenance windows, retry budgets, and emergency priority without starving repair. Observe desired/observed drift, version skew, upgrade stage age/failure, service target burn by cell, migration main rules, telemetry gaps, backup freshness/restore success, key errors, residency violations, and support access. DR is per tier: cell evacuation for shared, warm region or documented rebuild for dedicated, customer-coordinated VPC/air-gap. Quarterly restore proves RPO/RTO.

First standardize release version lists and conformance without changing deployment. Onboard shared internal cells. Add dedicated instances. Then VPC agents with two customers. Finally offline bundle/import/receipt workflows. Run fleet inventory and drift in read-only mode before automatic cross-checking. Prove backup restore and rollback for each mode. Publish support/version policy and error budgets jointly with customers.

The trade-off is fleet uniformity versus customer sovereignty. Forcing one version simplifies operations but violates change control and air-gap reality. Unlimited skew multiplies vulnerability and test matrices. I set explicit supported windows, compatible expand/contract contracts, strong incentives and visibility, while preserving customer approval and a narrow emergency process.

I’ll summarize write-once target version and settings, independent data planes, ringed upgrades, sovereignty, and recovery.