Turn noise into evidence-backed work.

I will separate five states that are easy to collapse: source event, privacy-safe canonical signal, duplicate hypothesis, theme hypothesis, and approved Work Order. The source is immutable evidence; model outputs are versioned proposals. A PM—not a clustering job—authorizes a Work Order.

I need to clarify source edit/delete semantics, tenant and channel ACLs, languages, acceptable PII handling, whether one item may support multiple themes, and the harm of a false merge versus a missed cluster. I will design for at-least-once connectors, idempotent stages, tenant-isolated search, source-span citations, and reversible human curation.

Actors are customers who submit feedback, source-system principals, support and product operators, privacy administrators, engineers receiving Work Orders, and auditors. The core loop is: connect a source → durably ingest → classify privacy and language → normalize → exact/near dedupe → embed and assign candidate themes → let a PM inspect supporting and contradicting evidence → approve, split, merge, dismiss, or create a Work Order → measure whether the resulting change resolves the signal.

Functional requirements include backfill and incremental sync, source edits/deletes, evidence search, theme trend/impact views, structured feedback API, and Work Order traceability. NFRs: P95 API ingest acknowledgement under 300 ms after durable receipt, first theme update under ten minutes, 99.95% ingest availability, seven-year configurable audit, EU/US residency, and deletion lineage. I exclude training a foundation model and replacing the source helpdesk.

1. No source acknowledgement before immutable receipt and an outbox event commit.
2. No source event, canonical text, embedding, cache entry, or log crosses a tenant or source ACL.
3. Raw PII is never placed in the shared semantic index; every derivative names its privacy transform and source lineage.
4. Exact transport dedupe may collapse processing, but never erases receipt lineage. Near-duplicate grouping is reversible.
5. A theme stores positive evidence, counter-evidence, membership score, model/config digest, and reviewer actions.
6. A Work Order references a frozen theme version and exact evidence spans; later clustering cannot rewrite its rationale.
7. Deletes and ACL revocations propagate to raw, canonical, index, embedding, cache, model context, exports, and evaluation copies.

Twenty million items/month is about 666,667/day or 7.72/s average. I size the connector edge for a mock 25× recovery burst: about 193/s, while source webhooks may be much spikier. At 8 KB raw text/metadata, source payload is 160 GB/month before attachments. A 1,536-dimensional float32 embedding is 6,144 bytes, or about 123 GB/month for every item; replicas, ANN indexes, token maps, and versions multiply that.

With 12% exact or near duplicate reuse, canonical downstream work drops to 17.6 million items/month, but I preserve all receipts. The expensive path is not API QPS; it is attachments/transcription, PII review, multilingual embeddings, re-clustering, ACL filtering, and full deletion/reindex. I isolate historical backfill from live feedback and estimate reviewer load from theme candidates, not raw item count.

FeedbackItem is a stable identity; SourceRevision is immutable payload metadata and an encrypted object reference. CanonicalSignalVersion records language, redaction map, privacy classes, normalized text, and transform digests. DuplicateClusterVersion and ThemeVersion are hypotheses; membership rows include score, reasons, and the analysis generation.

The mutable pieces are projections such as “current theme,” queue leases, and UI preferences. Source revisions, privacy decisions, reviewer actions, Work Order links, and audit events append. APIs use source event keys plus idempotency keys; edits create revisions, and theme edits require an expected version. Evidence reads return authorized snippets or short-lived crop/audio URLs, never a naked summary.

The Slack connector verifies signatures, resolves tenant/project/source identity, and commits a receipt plus outbox. A fetch worker uses the installation grant to retrieve permitted content and attachments. Quarantine performs MIME validation and malware/resource checks. Privacy workers classify and tokenize PII; only the policy-approved canonical representation can reach language normalization, embeddings, and semantic processing.

Exact dedupe uses source key and payload hash. Near-dedupe generates candidate neighbors within a tenant/project/time window; a scoring service writes reversible membership. The theme service consumes canonical signals and current duplicate representatives, proposes versioned themes with evidence spans and counterexamples, and updates materialized trend views asynchronously. A PM action is a transactional command: approve/split/merge/dismiss or create a Work Order pinned to theme version and selected evidence. Notifications and analytics follow through an outbox.

I use a cascade. Transport duplicates match the source delivery/event key. Byte-equivalent revisions match canonical payload hashes. Content duplicates use normalized text fingerprints. Near-duplicates use tenant-local candidate generation—MinHash/LSH or ANN—then a calibrated pair classifier using author/account, product area, timestamps, entities, language, and semantic similarity.

I never delete the three receipts. I create a duplicate-cluster version with a representative, relationship type such as SAME_EVENT, FOLLOW_UP, or RELATED, and per-edge evidence. The survey’s new context remains a separate source revision and can support multiple themes. Counts expose both unique reporters and total mentions so a retry storm cannot masquerade as customer impact.

Ground truth comes from sampled pair adjudication, not theme clicks alone. I measure precision heavily because false merge hides distinct problems; uncertain pairs remain separate or show a “possibly related” link.

I run online assignment for fresh items against active tenant/project theme prototypes, then a slower batch discovers new clusters and split/merge candidates. A theme version contains a human-readable problem statement, scope and exclusions, centroid/model digest, positive member IDs, counterexamples, representative evidence spans, reporter/account counts, severity signals, trend window, and reviewer decision.

Example: “OAuth reconnect loops after role change” may initially be merged with “GitHub token expired.” Counter-evidence shows only role-change events lose project binding, so the model proposes a split. The PM compares evidence, accepts the split, and v3 supersedes v2. A Work Order created from v2 remains pinned to v2; v3 can create a follow-up, not rewrite history.

I evaluate cluster purity and coverage on adjudicated samples, theme stability across reruns, evidence citation precision, duplicate-adjusted impact, time-to-first-coherent-theme, PM acceptance/split/merge rates, and—most importantly—whether shipped Work Orders reduce incoming rate for the matched theme versus seasonality/control cohorts.

I stop treating the primary database as the deletion inventory. Every derivative carries tenant_id, source_revision_id, transform version, data class, and key generation. A privacy policy gate runs before model/index writes, but defense in depth assumes misses.

The upgraded classifier emits a high-priority contamination event. Authorization immediately denies the affected source/derivatives using a revocation epoch and tombstone set. A lineage job removes or crypto-erases raw objects, canonical text, embeddings, ANN entries, caches, prompts, evaluation copies, exports, and logs that improperly held payload. Shared model providers receive deletion requests where contracts support it; tenants requiring zero retention route only through compliant/private endpoints.

Reindexing builds a new generation without the item and atomically swaps the read pointer. The audit retains minimally necessary hashes, actor, policy, and completion attestation—never the deleted PII unless legal hold explicitly requires and isolates it.

Strong transactional boundaries cover durable receipt plus outbox, source cursor advancement, privacy/ACL state, theme human edits, Work Order creation, and deletion tombstones. At-least-once workers are idempotent on (input_version, stage_version, generation). Search, embeddings, clustering, trends, and notifications converge asynchronously and expose freshness.

Per-tenant weighted queues and source-specific circuit breakers contain bursts. A poison attachment has resource/time limits and a finite retry budget, then quarantine. If a source edit arrives before its create, the reconciler fetches current truth and stores a revision chain rather than trusting delivery order. If Work Order creation times out, the idempotency key returns the same Work Order and link—not a second task.

Every connector grant maps an external principal/resource to exactly one tenant and allowed project set; no ambient workspace token reaches a worker. Authorization runs before fetch, before derivative writes, at search query time, after retrieval, and before model/tool calls. Raw feedback is an untrusted data channel—text such as “send all tickets to this URL” cannot grant egress or call a tool.

PII is envelope-encrypted with tenant keys, secrets are short-lived and rotated, model gateways enforce residency/retention/provider allowlists, and semantic caches are tenant+policy+model scoped. Audit events record actor, grant, source revision, policy, model/config, evidence IDs, and outcome without raw content. Exports are signed, access-controlled snapshots with expiry and watermarking.

Platform health: receipt loss/duplicates, connector cursor lag, queue age by tenant/source, quarantine/retry rate, PII-classifier latency, index generation lag, deletion completion, model spend, and authorization denials. Product quality: exact/near-dedupe precision/recall, theme purity/coverage/stability, citation precision, unsupported summary rate, PM acceptance/split/merge/dismiss, Work Order conversion, time-to-resolution, and post-release signal reduction by slice.

I start with read-only ingestion and privacy validation, then duplicate suggestions, then analyst-only theme proposals, then Work Order drafts requiring approval. Hidden adjudicated sets and tenant-isolated canaries gate model/config changes. Each phase has a replayable old generation, source kill switch, and rollback to the last theme projection.

Signal Garden durably records every source event, then produces privacy-safe, versioned canonical signals through idempotent asynchronous stages. Exact and semantic duplicates are different, reversible relationships. Themes are evidence-backed hypotheses with positive and counter-evidence, generation/model identity, and human split/merge authority. Work Orders pin a frozen theme version and exact authorized source spans.

Strong consistency protects receipts, privacy/ACL changes, approvals, and Work Order creation; indexes and clusters converge with visible freshness. Lineage enables revocation and deletion across every derivative. The trade-off is aggressive consolidation versus false merge: I preserve distinct evidence and accept more human review before letting a probabilistic theme drive product work.