REPOSITORY OBSERVATORY / MOCK 03

Code Intelligence & Drift Control

Design a multi-tenant platform that continuously maps many repositories into searchable symbols and detects when implementation stops matching approved intent.

30 MINUTES 16 INTERVIEWER TURNS FULL Q & A INTERACTIVE LAB

The problem on the whiteboard

Read this once, then begin aloud. Spend the first two minutes establishing contract, actors, risk and what must remain authoritative.

Design CodeScope, a multi-tenant code-intelligence and drift platform. It connects read-only GitHub and GitLab repositories, incrementally indexes code across branches, exposes symbol-aware search, links code to approved requirements and blueprints, and comments on pull requests when a change appears to violate intent. A finding must carry enough evidence for a human to reproduce and dismiss it.
Central design thesis

The key decision is to separate authoritative artifacts and repository snapshots from derived indexes and probabilistic drift findings. Publication and permissions are strongly controlled; search and drift are rebuildable, version-labeled projections.

Scope before components

These are supplied mock numbers for sizing and follow-ups. Say your assumptions before using them.

Portfolio 250 tenants · 8,000 repositories · 1.2B lines indexed
Change load 25,000 push events/day · 15× burst after workday start
Freshness 95% of changed symbols searchable in < 5 min; PR drift result < 7 min
Queries 400 searches/second at peak; 95% of keyword searches < 250 ms, meaning-based searches < 700 ms
Trust 99.9% query availability; no cross-tenant result; every finding reproducible

Actors

  • Engineer searching and reviewing a PR
  • PM or architect maintaining approved intent
  • Coding agent requesting scoped context
  • Tenant admin connecting repositories
  • GitHub/GitLab webhook and content APIs
  • Security or compliance auditor

Functional scope

  • Read-only repository onboarding and revocation
  • Incremental parse, symbol graph, lexical and semantic search
  • Trace links among artifact revisions, commits and symbols
  • Drift analysis with PR evidence and dismissal feedback
  • Tenant-aware freshness, audit, metering and backpressure

Explicitly out

  • Generating or merging code
  • Replacing the Git host
  • Compiling every language perfectly in v1
  • Proving semantic equivalence
  • Autonomous blocking of production merges

A traceable end-to-end path

Name the authoritative state, derived projections, asynchronous boundaries and the exact point where a business decision becomes durable.

Code Intelligence & Drift Control reference architecture Six stage architecture from Git host through Drift review. Git host signed events Webhook log dedupe + order Snapshot fetch immutable commit Parse + graph AST / symbols Search indexes lexical + vector Drift review evidence, not verdict audit · policy · metrics · lineage
01 Git host signed events
02 Webhook log dedupe + order
03 Snapshot fetch immutable commit
04 Parse + graph AST / symbols
05 Search indexes lexical + vector
06 Drift review evidence, not verdict

Keys that preserve history

The table is logical, not a mandate for one database. Choose physical stores after access patterns, transactions, retention and rebuildability are clear.

Entity Primary / unique key Important fields and purpose
Repository (tenant_id, repo_id) provider_installation, encrypted_locator, default_branch, permission_epoch, state
CommitSnapshot (repo_id, commit_sha) parent_shas, observed_at, manifest_hash, index_state
FileVersion (repo_id, commit_sha, path_hash) blob_sha, language, parser_version, content_acl
Symbol (repo_id, commit_sha, symbol_id) qualified_name, kind, span, signature_hash
SymbolEdge (repo_id, commit_sha, from_id, type, to_id) calls/imports/implements/reads; extractor_version
ArtifactRevision (tenant_id, artifact_id, revision) approved_body_hash, schema_version, approval_state
TraceLink (artifact_revision, symbol_id, commit_sha) link_type, creator, confidence, evidence
DriftFinding (tenant_id, finding_id) analysis_run, rule/model versions, evidence spans, state, reviewer
POST /v1/repositories

Connect selected repo; idempotency-key required

202 {repoId,indexRunId}
POST /v1/webhooks/{provider}

Verify signature; persist raw envelope before ack

202
GET /v1/search?q=&repo=&commit=&cursor=

Hybrid symbol/code search pinned to permission epoch

200 {hits,nextCursor,indexedThrough}
POST /v1/trace-links

Propose requirement-revision ↔ symbol-at-commit link

201 / 409 stale revision
POST /v1/pulls/{id}/analyze

Deduped by repo + head SHA + policy bundle

202 {analysisRunId}
POST /v1/findings/{id}/reviews

Confirm, dismiss, suppress, with reason and etag

200 {state,version}

The complete mock interview

Answers are written in a speakable first-person style. Turn on Practice Mode to hide them, answer aloud, then reveal one at a time.

INTERVIEWER

Design the system. Where do you start?

CANDIDATE

Candidate · clear top-down answerI would start by defining what we can trust. GitHub or GitLab owns the code. Approved requirements own the intent. Our system builds fast search and drift warnings from those two sources.

Why this mattersThis gives the interviewer a map before any boxes appear, and it makes the system's most important trust decision explicit.

INTERVIEWER

Clarify requirements and the first release.

CANDIDATE

Candidate · clear top-down answerFor version one, teams connect repositories, search code, link requirements to code, and see evidence-backed warnings on pull requests. We will not write code or block merges yet. That keeps the first release useful and safe.

Why this mattersA narrow first release keeps the design realistic. It also shows which tempting features can wait without weakening the core product.

INTERVIEWER

Do a back-of-the-envelope estimate.

CANDIDATE

Candidate · clear top-down answerThe average traffic is small, but pushes arrive in bursts and the first import is huge. For 80,000 repositories with one million files each, the first scan can cover tens of billions of files, while a normal push may change only a few dozen. I would keep imports in a slower queue and reserve workers for live pull requests. The math tells us scheduling and reuse matter more than raw parser speed.

Why this mattersThe calculation is not about perfect forecasting. It identifies the part most likely to run out of capacity or delay users.

INTERVIEWER

Show your data model and access patterns.

CANDIDATE

Candidate · clear top-down answerThe main records are repository, commit, file, symbol, requirement version, link, and finding. We never change an old commit or approved requirement. A branch pointer simply tells us which completed commit is current. The common reads are search by tenant and commit, symbols by file, links from one requirement version, and findings for one pull request. Those access paths drive the indexes.

Why this mattersA small set of records is easier to reason about. Exact versions keep later changes from silently rewriting an earlier decision.

INTERVIEWER

What APIs matter?

CANDIDATE

Candidate · clear top-down answerWe need simple endpoints to connect a repository, start indexing, search a chosen commit, request pull-request analysis, and review a finding. Long jobs return an operation ID so the client can check progress instead of holding one HTTP request open.

Why this mattersThe API follows the user journey and makes slow work visible. A client should never hold one request open for a long job.

INTERVIEWER

Walk one push end to end.

CANDIDATE

Candidate · clear top-down answerA signed webhook says a push happened. We save its delivery ID before acknowledging it, then ask the code host for the real current commit. We fetch only changed files, update the code map, run drift checks, and publish the completed search view with one pointer switch. Users either see the old complete view or the new complete view, never a half-built mix.

Why this mattersTracing one real request proves the boxes connect. It also exposes where we acknowledge success and where a retry may begin.

INTERVIEWER

Deep dive: how is incremental indexing both fast and correct?

CANDIDATE

Candidate · clear top-down answerTo stay fast, we reuse files whose content hash did not change. To stay correct, we also invalidate callers when a public symbol or build setting changes. We build a new index for each commit and switch readers to it only when complete. If a parser cannot prove that reuse is safe, it rebuilds the wider dependency group. That spends more compute to avoid a silently wrong index.

Why this mattersThe deep dive starts only after the basic path is clear. Use one concrete example, then state the speed-versus-correctness choice.

INTERVIEWER

Deep dive: what exactly is drift?

CANDIDATE

Candidate · clear top-down answerA drift warning means code may no longer match an approved requirement. Cheap exact checks run first. An AI model may help with unclear meaning, but it must point to exact requirement text and changed code lines. A person still decides.

Why this mattersEvidence must be visible to a reviewer. AI may help rank or explain, but it does not become the final authority.

INTERVIEWER

How do consistency and idempotency work?

CANDIDATE

Candidate · clear top-down answerA few small records need immediate agreement, such as access and review decisions. Search can be a little behind, but every response shows which commit it covers. Repeated webhooks and jobs are safe because each has a stable key.

Why this mattersUse immediate agreement only for decisions that cannot be ambiguous. Let rebuildable views catch up and always show their freshness.

INTERVIEWER

Failure injection: duplicate webhooks arrive out of order, then a force-push removes commits. Revise.

CANDIDATE

Candidate · clear top-down answerI never trust webhook arrival order. I ask the code host for the current branch head and build from commit IDs. After a force-push, old data can remain for audit, but users only see the new branch view.

Why this mattersA failure answer should name the uncertain state first. Then explain how the system learns the truth before it repeats an action.

INTERVIEWER

How do you handle backpressure and giant monorepos?

CANDIDATE

Candidate · clear top-down answerLarge repositories get smaller chunks and cost-based scheduling. Live pull requests get reserved capacity, while imports and full rebuilds use a separate pool with per-tenant limits. If ten pushes arrive quickly, we may skip intermediate branch views and build the newest commit, while still keeping every commit addressable. Queue age, not queue length alone, triggers scaling and admission limits.

Why this mattersBusy tenants and large jobs need limits. Reserved capacity keeps one customer or one import from delaying everyone else's live work.

INTERVIEWER

Failure injection: access is revoked halfway through indexing. What changes?

CANDIDATE

Candidate · clear top-down answerRevoking access must hide the repository at once. We raise an access version number, stop new fetches, cancel queued work, and reject any worker trying to publish results created with the old access version.

Why this mattersRemoving access must affect reads and in-flight workers. Cached data is never a reason to keep showing revoked information.

INTERVIEWER

Security and tenant isolation?

CANDIDATE

Candidate · clear top-down answerEvery record, object path, queue message, index document, and cache key includes the customer ID. Workers use short-lived repository tokens limited to one job. Parsers run in isolated sandboxes with no open network access. Search checks permission before retrieval and again before returning results, so a stale index filter cannot leak revoked code. Logs contain IDs and timings, not source text.

Why this mattersSecurity belongs in keys, tokens, workers, and logs. It is not a single box added to the diagram at the end.

INTERVIEWER

What do you measure, and how do you roll this out?

CANDIDATE

Candidate · clear top-down answerI measure how long a push takes to become searchable, queue age, search speed, access-removal time, and drift accuracy. I would launch read-only search first, then advisory warnings, then optional merge checks only after teams trust the results.

Why this mattersMetrics should cover user speed, system safety, and result quality. Rollout starts read-only and earns stronger automation with evidence.

INTERVIEWER

Name trade-offs and traps.

CANDIDATE

Candidate · clear top-down answerThe main trade-off is speed versus certainty. Reusing old work is fast, but uncertain dependency changes require a wider rebuild. Hybrid exact plus semantic search is better than vectors alone because code identifiers and evidence matter.

Why this mattersEvery design choice has a cost. Name the simpler alternative and explain which requirement makes the added complexity worth it.

INTERVIEWER

Give me your one-minute recap.

CANDIDATE

Candidate · clear top-down answerIn short: save commit snapshots that never change, reuse unchanged files, publish complete search views, and show freshness everywhere. Drift starts with exact evidence, uses AI only for unclear cases, and stays advisory until measured quality supports stronger action.

Why this mattersThe recap should repeat the user flow, the main trust rule, the hardest failure, and the most important trade-off—not every component.

Two places to earn the strong hire

These mechanisms are the interview’s differentiators. Be able to redraw each from memory and defend its failure behavior.

01

Deep dive A · Versioned index publication

A branch name is mutable; a commit is not. Build immutable commit segments, then publish one small branch pointer only after the required segment set is complete.

  1. Manifest diff identifies changed blobs
  2. Parser namespace isolates extractor versions
  3. CAS prevents an older run overwriting a newer head
  4. Lexical, graph and semantic freshness can advance separately
  5. Reconciler compares host refs with projections
02

Deep dive B · Evidence ladder for drift

Move from high-precision deterministic facts toward probabilistic semantic judgments, preserving the input manifest at every rung.

  1. Broken trace or API contract
  2. Impacted symbol neighborhood
  3. Retrieved approved clauses
  4. Bounded model classification with citations
  5. Human review and suppression feedback

Revise, do not hand-wave

When the interviewer injects a fault, restate the violated assumption, change state or protocol, and name the new invariant.

Injection Recovery mechanism Tempting wrong answer
01Webhook replay/out of order Provider delivery dedupe + resolve canonical DAG Duplicate business effect
02Force-push New projection generation; keep orphan snapshots per retention Old head shown as current
03Parser regression Versioned namespace, canary, rollback projection Corrupting only good index
04Provider rate limit Token bucket, jitter, live/backfill lane isolation Retry storm
05Repo revoked Permission epoch fence, synchronous hide, async purge Data survives in vector/cache
06Model unavailable Serve deterministic checks; mark semantic analysis pending Blocking PR workflow

Calculate before you provision

Use orders of magnitude to expose the bottleneck. State what this simple model omits.

Index freshness lab

Change mock workload and see whether the live indexing lane can meet the five-minute target.

Estimated raw parse time — Change an input to recalculate.
Fetch
—
Parse
—
Graph
—
Publish
—

Prove quality in production

A system is incomplete without observable user outcomes, staged deployment, rollback and an answer to “how will we know?”

Scoreboard

  • Head-to-search lag p50/p95/p99
  • Search p95 and zero-result rate
  • Finding precision by class
  • Reviewer confirmation and suppression
  • Permission-fence violations (target zero)
  • Cost per changed KLOC

Rollout ladder

  1. 01 Exact symbol search on two languages
  2. 02 Incremental indexing + visible freshness
  3. 03 Silent deterministic drift evaluation
  4. 04 Dashboard-only mixed findings
  5. 05 Opt-in capped PR comments
  6. 06 Tenant-specific calibration and expansion

Corrections worth memorizing

Use these as flash cards. The right column is the compact sentence you want available under pressure.

Trap Better move
Calling a branch authoritative Pin snapshots and links to commit SHA.
One giant mutable index Publish immutable versioned segments.
Vector search for identifiers Hybrid lexical, symbol and semantic retrieval.
Trusting webhook order Reconcile against the host commit graph.
“Exactly once” hand-wave At-least-once transport + idempotent effects.
No revocation story Permission epochs plus complete derived-data purge.

Vocabulary without fog

Define the term, then connect it to a concrete invariant in this design. Avoid dropping vocabulary as a substitute for reasoning.

AST
Abstract syntax tree: a parser’s structured representation of source.
Symbol
Named program element such as a class, function, type or constant.
Projection
Rebuildable current view derived from immutable source events/snapshots.
Permission epoch
Monotonic version fencing caches and workers after access changes.
Drift
Evidence-backed mismatch hypothesis between approved intent and implementation.
Content addressing
Identifying bytes by a cryptographic hash to deduplicate and verify them.
CAS
Compare-and-swap: update only if the expected version is still current.
Reconciliation
Periodic comparison of desired/canonical state with derived state.

Close with a decision, not a component list

CodeScope stores immutable repository snapshots and approved artifact revisions, then builds versioned, tenant-fenced search and graph projections. Signed webhooks feed an at-least-once, cost-aware pipeline; content hashes and commit keys make it idempotent. Search always reports commit and freshness. Drift begins with deterministic evidence, uses models only as a reproducible classifier, and keeps humans authoritative. Permission epochs stop revoked data immediately; reconciliations repair missed events. I would ship search, shadow drift, dashboard findings, then carefully calibrated PR comments, measuring both SLOs and reviewer trust.

Rehearse again