Connect everything. Trust nothing implicitly.

I will design two planes. The connection control plane owns OAuth handshakes, encrypted refresh tokens, external resource discovery, project bindings, policy, revocation epochs, and audit. The invocation data plane accepts a signed internal request, re-authorizes subject + actor + project + resource + action, exchanges it for a short-lived audience-bound capability, invokes the connector, validates the response, and records a receipt.

The key invariant is that a connector credential is never ambient worker state. A Slack message, retrieved document, model output, or MCP server cannot choose a different tenant/project or expand permissions. Every hop is derived from server-side bindings and explicit delegation.

Actors include organization admin, project admin, human user, Slack user, coding agent, automation service account, connector broker, external provider, MCP server, and auditor. I clarify whether connections are user- or organization-owned; read versus write actions; project/resource binding; provider installation behavior; SSO/SCIM expectations; token residency; and whether MCP servers are trusted code or untrusted third parties.

Primary journeys are connect/discover/bind, user-triggered read, agent-triggered write with approval, webhook intake, refresh/rotation, scope change, revoke/uninstall, incident kill switch, and audit export. Out of scope: implementing GitHub/Slack themselves or a general identity provider. The gateway must expose provider-neutral contracts while preserving provider-specific semantics.

• OAuth authorization code + PKCE/state/nonce, admin-consent paths, resource discovery, and explicit project bindings.
• Per-user, service, project, resource, operation, environment, and purpose scopes; policy approval for consequential writes.
• Webhook verification/dedupe/reconciliation; async credential refresh and rotation; immediate emergency revoke.
• MCP tool discovery with pinned schemas, allowlists, input validation, output size/type limits, and no tool-chosen credentials.
• Delegation chain and immutable invocation receipts: initiator, actor, grant, resource, action, approval, token audience, result.

Invariants: external identity is mapped server-side; tenant/project comes from trusted routing, not user/tool payload; capabilities are short-lived and audience-bound; refresh tokens stay only in a hardened vault/broker; revocation blocks new calls even if indexes/caches lag; and response data is classified and authorized before it enters agent context.

Thirty million tool calls/day is 347 calls/s average. At a mock 20× peak that is 6,944/s. If 90% of calls validate a locally signed capability and a fresh policy epoch, only about 694/s need a control-store authorization read; every call still performs cryptographic validation, resource/action matching, rate limiting, and audit buffering.

At 1.2 KB per invocation receipt, calls generate about 36 GB/day before indexes/replicas. Twenty-five thousand webhooks/s at 2 KB envelope is 50 MB/s, but provider fetch and repository indexing dominate. I durably accept verified webhooks, dedupe by provider delivery ID, coalesce by resource/ref, and reconcile current provider state asynchronously. OAuth callbacks and refreshes stay off the hot invocation path.

ExternalConnection identifies provider tenant/installation and vault reference; the refresh token lives only in an HSM/KMS-backed secret broker. ResourceBinding maps provider resource IDs to one internal tenant/project. GrantVersion stores subject, allowed actions/resources, consent, policy, and revocation epoch. Delegation adds the workload actor, purpose, approval, audience, and expiry. InvocationReceipt records request hash and provider correlation—not raw secrets.

The user-facing API starts OAuth with signed state tied to browser session, tenant, requested connection kind, and expiry. The callback verifies state/PKCE/issuer and stores the token in the vault before resource selection. Internal workers never ask for refresh tokens; they call token exchange for one tool/resource/action. Revocation increments epochs transactionally, disables bindings, and emits a high-priority invalidation event.

Slack ingress verifies the workspace signing secret, delivery ID, channel/thread, and mapped Slack user. It records an event and resolves the project binding server-side. The agent orchestrator receives a context envelope naming subject, actor/run, tenant/project, purpose, and requested capabilities—not credentials.

For the GitHub read, the policy decision point checks the user/project/resource/action and current grant epochs, then the token broker mints an internal capability for the GitHub connector. The connector retrieves only the bound repo/commit/path and returns classified data through an output policy check. For ticket creation, the MCP catalog supplies a pinned tool schema; the policy layer requires write approval, validates arguments and target project, and gives the MCP proxy a one-call capability. An idempotency key and receipt make an ambiguous timeout reconcilable.

The message is untrusted data and cannot select the connection, tenant, project, or scope. Project resolution comes from the signed workspace/channel/thread binding. The authorization tuple is (subject=Slack user mapping, actor=agent run, tenant, project, connector, resource, action, purpose, approval, grant epochs). All fields except user intent are server-derived.

The policy engine intersects—not unions—human permission, project binding, connection installation scope, workload capability profile, action policy, and data classification. The resulting capability has GitHub connector as audience, exact repo/path or query constraint, read-only action, 120-second TTL, run/step ID, and epochs. The connector rejects resource IDs in the request that are outside the token. A separate post-retrieval check prevents a broader provider response from leaking.

The audit displays “Alice via Slack thread S, agent run R, project P, read repo X/path Y under grant G”—both subject and actor. Denial returns a generic message without revealing the repository exists.

Revocation is an authoritative transaction: mark connection/grants disabled, increment connection and affected project epochs, revoke vault access, append audit/outbox, and stop refresh. Gateways subscribe to the high-priority invalidation stream and maintain a small deny set. Every connector invocation, pagination fetch, artifact publish, and privileged checkpoint compares the token epochs with current or safely cached epochs.

Existing jobs move to REVOKED_PENDING_CLEANUP; workers may finish local computation on already authorized bytes only if policy allows, but cannot fetch more or publish a new shared index generation. Search authorization rejects results regardless of stale index content. Derived-data retention follows tenant policy: delete/crypto-erase source content and indexes, or preserve encrypted snapshots under legal hold with access removed.

If the epoch service is unreachable, reads of sensitive data and all writes fail closed. Low-risk public metadata could use explicitly configured grace, but never by accidental cache behavior.

Connection onboarding snapshots tool names, JSON schemas, risk classes, egress domains, data residency, and required scopes. An admin approves a versioned catalog; schema drift disables or quarantines the changed tool until reapproval. Agents can request catalog tool IDs, not arbitrary endpoints.

The MCP proxy validates arguments, size, types, resource bindings, and idempotency before invocation. It sends a one-call capability; no platform refresh token or unrelated secrets enter the server. Responses are size/time bounded, malware/content-classified, and wrapped as untrusted tool data. Returned text cannot automatically trigger another tool; each next call is a fresh policy decision, and sensitive actions need human approval.

Network egress is allowlisted, DNS/IP rebinding is defended, server certificates/identity are pinned where possible, and tenants may route through a private proxy. Tool-call traces store schemas, hashes, decision IDs, and receipts while redacting secret/raw payload fields.

Token rotation is a compare-and-swap vault operation with credential version. The broker stores the new token encrypted before marking the old version retired; if provider semantics invalidate the old token immediately, the connection enters REFRESH_UNKNOWN and a single-flight reconciler tests or reauthorizes—workers never fan out refresh attempts.

For the ticket, the tool gateway first commits EffectIntent(effect_key, request_hash, REQUESTED). It passes the stable key if the provider supports idempotency. On timeout, state is UNKNOWN; recovery queries by key/correlation or creates a human reconciliation task. No blind retry. If reconciliation finds success, record external ID and hash; if not found and policy allows, retry with the same effect key.

Connector-specific adapters expose these semantics explicitly. A generic “retry three times” middleware is unsafe for rotating credentials and non-idempotent writes.

Ingress verifies and durably buffers webhooks, then per-provider/installation queues apply quotas and coalescing. Provider circuit breakers stop futile calls; exponential backoff honors rate-limit resets. Hierarchical fairness reserves interactive tool capacity, caps backfill/indexing, and meters per tenant/project/connector. Queue messages contain opaque IDs—not tokens or fetched source.

I track connection health, refresh success, webhook dedupe/lag, policy latency/cache age, revocation propagation, denied confused-deputy probes, token exchange volume, tool schema drift, ambiguous effects, provider rate limits, tenant fairness, and audit receipt gaps. Synthetic canary tenants continuously attempt cross-project reads and revoked-token use. Secrets never appear in application traces.

Multi-region control uses one authority per grant shard or consensus for grant/epoch writes; data-plane validation can run regionally from signed policy snapshots plus revocation stream. On uncertainty, privileged access fails closed.

Start with synthetic providers and conformance suites: state/PKCE replay, redirect abuse, tenant/resource confusion, epoch revocation races, scope downgrade/upgrade, schema drift, rate limits, token rotation, pagination, response overreach, ambiguous effects, SSRF, prompt injection, and secret redaction. Property tests assert that adding untrusted request fields can never expand authority.

Roll out read-only resource discovery, then selected read actions, then sandboxed write proposals, then approved writes per connector/action. Each connector has a kill switch, minimum version, scoped canary tenants, incident playbook, and audit export. We measure successful authorized outcomes—not raw call success—and block promotion on any cross-tenant or post-revocation access.

The Switchboard keeps OAuth refresh credentials in a hardened broker and maps external installations/resources to one tenant/project in an authoritative control plane. Every tool call re-derives authorization from subject, workload actor, resource, action, purpose, approval, and revocation epochs, then exchanges for a short-lived audience-bound capability. Connectors enforce resource constraints again and classify outputs before agent context.

Webhooks and jobs are durable, idempotent, fair, and reconcilable; revocation fences fetch, publish, and search independent of index lag; ambiguous writes use effect receipts instead of blind retry. The trade-off is cache performance versus revocation certainty: locally verified capabilities are fast, but privileged steps must consult a fresh epoch or fail closed.