Code Intelligence & Drift Control

I will first pin the trust boundary and success measure. GitHub and GitLab remain the source of repository bytes; approved requirement and blueprint revisions remain the source of intent. Our indexes and findings are derived. The user journey is connect → index → search → change → drift review. I will design for deterministic provenance around a probabilistic classifier, not pretend the classifier is truth. I also want to know whether a drift finding blocks a merge.

V1 supports selected branches for five major languages, exact and semantic search, approved artifact-to-symbol links, webhook-driven incremental indexing, and PR findings with evidence. Admins can revoke a repository. A reviewer can confirm, dismiss, or suppress a finding and that decision becomes evaluation data. I exclude code writes, merge gating, full build execution, and semantic equivalence proofs. I would ask about retention after unlink; my default is immediate query revocation and a configurable asynchronous purge.

• Hard invariant: tenant and repository authorization is checked before retrieval and again after ranking.
• Product truth: every result states commit SHA and index freshness.
• Safety: no autonomous merge blocking until calibrated per tenant.

The initial 1.2B LOC is a bulk backfill, so isolate it from the live lane. At a mock effective 20,000 LOC/s per parser worker, 100 workers parse 2M LOC/s: about 10 minutes of pure parse time, but fetch, language variance and graph writes dominate, so I budget hours and expose progress. The steady state is modest on average: 25,000 pushes/day is 0.29/s, but a 15× burst and large monorepos make per-event averages misleading. I size queues by changed bytes and files, not event count. Search at 400 QPS needs independently scalable replicas.

25,000 / 86,400 ≈ 0.29 pushes/s average
Burst ≈ 4.3 pushes/s
1.2B LOC / (100 × 20k LOC/s) ≈ 600 s raw parse floor

Every repository record is tenant-scoped. CommitSnapshot is immutable and keyed by repository plus commit SHA. FileVersion references content by blob SHA so unchanged files deduplicate. Symbols are immutable within a commit; stable lineage is a separate mapping because a rename can break identity. ArtifactRevision is append-only after approval. TraceLink pins both an artifact revision and a symbol at a commit, never a moving branch. DriftFinding records the complete analysis manifest: input hashes, parser/rule/model versions, evidence spans and decision.

• Search by tenant + authorized repo set + query + commit/default-branch projection.
• Incremental index by repo + commit; compare parent manifest to enumerate changed blobs.
• Audit by finding ID → analysis run → exact artifact revision and code spans.

Repository creation and pull analysis are asynchronous and return durable operation IDs. Webhook ingestion acknowledges only after the signed raw envelope is durably persisted. Search accepts a commit or a branch projection and returns indexedThrough so clients can distinguish no match from stale index. Mutations use idempotency keys and optimistic etags. Review endpoints require a reason because dismissals are product signal, not just UI state.

The edge verifies provider signature, installation ID and replay window, stores the envelope with provider delivery ID, then returns 202. A sequencer resolves the commit DAG rather than trusting webhook arrival order. The snapshot worker fetches changed manifests and blobs using a short-lived provider token, hashes all inputs, and publishes a CommitSnapshot. Language workers parse changed files in sandboxes, update symbol and edge segments, then atomically advance the branch IndexProjection from the parent commit to the new commit. Search fans out only to segments visible under the request permission epoch. Drift workers compare the PR diff and impacted symbol neighborhood against approved artifact revisions, run deterministic candidates first, then a bounded model judge, and persist evidence before posting one summary comment.

• Control plane: tenants, connections, policies, operations.
• Data plane: fetch, parse, index, analyze.
• Reconciliation periodically compares host heads with branch projections.

I use content-addressed blobs and immutable per-commit segments. For a normal commit, diff the manifest against indexed parents, parse changed blobs, recompute local symbols, then invalidate dependent edges using the prior reverse-dependency graph. A merge commit may require both parents; if ancestry is missing or a force-push occurs, mark the projection degraded and schedule a bounded subtree or full rebuild. Publishing is a compare-and-swap on branch projection: only point search to commit C after all required lexical, symbol and ACL segments for C are ready. Semantic embeddings may lag but advertise their own freshness.

• Never mutate the only good index in place.
• Use parser-version namespaces so a bad release can roll back.
• Large generated/vendor paths follow explicit policy and are visibly excluded.

Drift is a typed, evidence-backed hypothesis. First, deterministic checks find broken contracts, removed referenced symbols, missing acceptance-test IDs or changed API schemas. Second, impact expansion retrieves the smallest symbol and artifact neighborhood. Third, a model may classify a semantic mismatch, but it must cite artifact clauses and diff lines. A calibrator thresholds per finding class and tenant; low-confidence cases are omitted or shown only in a dashboard. Dismissal reasons feed an offline labeled set. The PR comment is capped and links to a review page so the bot does not spam.

• Precision matters more than recall for intrusive PR comments.
• Evaluate by class, language and repository—not a single aggregate score.
• Re-run must pin prompt, model snapshot when available, retrieval manifest and temperature.

Strong consistency is reserved for repository authorization, permission-epoch changes, artifact approval, branch projection publication, finding review and billing. Derived search indexes are eventually consistent and disclose freshness. Provider delivery ID dedupes webhooks; index work is keyed by repo + commit + parser bundle; analysis is keyed by repo + PR head SHA + policy bundle. Every stage writes outputs before committing its state transition, so at-least-once delivery is safe. A unique constraint and compare-and-swap close race windows.

I do not sequence by arrival time. I ingest all valid envelopes, query the host for canonical refs, and construct the commit DAG. Duplicate delivery IDs are no-ops; a different event for an already indexed SHA can update observation metadata without rebuilding. On force-push, create a new branch projection generation and stop advertising the orphan head. Immutable orphan segments can remain for retention/audit but are unreachable from current search unless explicitly pinned. A reconciler detects gaps or missed webhooks by polling branch heads. The UI shows indexing state rather than serving the old head as current.

Queues carry cost estimates such as changed bytes, file count and language, not only FIFO messages. Separate live PR, normal push and backfill pools with weighted fair scheduling per tenant. Cap concurrent fetches per provider installation to respect API limits. Chunk monorepos by dependency-aware directory partitions, checkpoint manifests, and coalesce superseded branch pushes while preserving every requested audit snapshot. Search capacity is isolated from indexing. If the seven-minute PR target is at risk, run deterministic checks on the diff first and label deeper semantic analysis pending.

• Per-tenant token buckets prevent a single backfill from starving others.
• Dead-letter only after bounded retries; retry from the last immutable checkpoint.
• Circuit breakers protect provider and model APIs.

Revocation increments the repository permission epoch in the strongly consistent control store. Query authorization immediately excludes the repo, caches key by permission epoch, and workers check the lease epoch before publishing. The fetch token is revoked and queued jobs are cancelled. Any output computed under an old epoch is quarantined and cannot advance a projection. Then a deletion workflow tombstones index segments, vector entries, cached snippets and model-context artifacts according to policy; an audit record proves completion. Logs retain identifiers and hashes without source content where regulation permits.

Use provider-app installation tokens with least privilege and short lifetime; never persist user PATs in workers. Encrypt repository locators and source at rest with tenant-scoped keys. A policy service returns authorized repo IDs and permission epoch; retrieval pushes that filter into lexical, vector and graph queries and post-filters results. Sandboxed parsers have no network and strict CPU/memory limits because repositories are adversarial input. Strip secrets before model calls, support provider allowlists or private inference, and defend against prompt instructions embedded in code by treating code as data. Audit admin connections, searches, exports and agent context assembly.

• Isolation tests deliberately plant canary symbols across tenants.
• Logs never contain raw source by default.
• Signed outbound PR comments bind repo, PR and analysis run.

Platform SLIs include webhook durability, head-to-search lag, queue age by lane, index publish success, query latency/availability, authorization denials, purge age and cost per changed KLOC. Quality uses a blinded labeled set: precision/recall by finding class, reviewer confirmation rate, time to decision, dismissal reason, comment suppression and reproducibility success. Roll out search first, then silent drift shadowing, then dashboard-only findings, then opt-in PR comments with a per-repo kill switch. Start with high-precision deterministic classes and expand after calibration.

• Alert on freshness SLO burn, permission-fence violation, model drift and reviewer complaint rate.
• Canary parser bundles on sampled repos before projection promotion.
• Compare confirmed bugs found against reviewer fatigue.

A graph database is not automatically required; commit-local adjacency segments plus a relational control plane may be simpler. Vector search alone loses exact identifiers and provenance, so hybrid retrieval wins. Reindexing the world on every push misses the incremental requirement. Mutating indexes in place makes rollback unsafe. Treating model output as drift truth creates false-positive fatigue. Hiding index freshness lies to users. Finally, unlinking a repo must have explicit visibility and deletion semantics—disconnect is not the same as purge.

CodeScope stores immutable repository snapshots and approved artifact revisions, then builds versioned, tenant-fenced search and graph projections. Signed webhooks feed an at-least-once, cost-aware pipeline; content hashes and commit keys make it idempotent. Search always reports commit and freshness. Drift begins with deterministic evidence, uses models only as a reproducible classifier, and keeps humans authoritative. Permission epochs stop revoked data immediately; reconciliations repair missed events. I would ship search, shadow drift, dashboard findings, then carefully calibrated PR comments, measuring both SLOs and reviewer trust.