Research snapshot: August 15, 2026
Prepared for: a full-stack/distributed-systems interview with 8090
Scope: public information only; this is not inside information and the
mock questions are predictions, not leaked interview questions.
This document contains the two flagship written mocks that follow directly from the research synthesis. The expanded companion suite contains 20 complete 30-minute HTML interviews covering the broader set of high-signal 8090 domains.
Companion visual rehearsals:
The simplest accurate model of 8090 is: one software platform plus one managed delivery business.
The company’s distinctive thesis is not merely that an LLM can generate code. It is that an enterprise needs enough context, provenance, deterministic validation, human approval, and operational ownership to trust the result. That thesis appears consistently in the home page, Software Factory product page, Enterprise offering, product documentation, X posts, customer stories, public challenges, and hiring descriptions.
The highest-signal interview themes are therefore:
The two flagship mocks embedded at the end target the strongest two combinations of those signals:
To avoid turning marketing into fact, this report uses four evidence classes:
Research included:
mib-doc-challenge
and top-coder-challenge;The research cutoff matters: Software Factory is changing rapidly, and the public changelog reached version 0.47.0 on August 11, 2026.
The legal entity named in the public terms is 8090 Solutions, Inc. The company also appears publicly as 8090 and, in some press, 8090 Labs. It was founded in 2024. A separate Chamath-controlled company’s SEC prospectus describes 8090’s aim as an AI-enabled factory for high-quality, maintained enterprise software and distinguishes it from “vibe coding.” That filing is useful disclosure, but it is not an 8090 securities filing.
The original 2024 public pitch was roughly to recreate 80% of established enterprise-software functionality at 90% lower cost. The name “8090” comes from that idea. The current positioning is broader and more operationally serious: build software faster, but make decisions, quality, traceability, deployment, and maintenance part of the product. Chamath has also publicly acknowledged that the literal original 80/90 promise may not be the final economic result.
In June 2026, 8090 announced $135 million of Series A financing, described on parts of its site as inclusive of seed funding. Salesforce led, with WndrCo, Craft Ventures, The Production Board, LAUNCH, and individual investors participating. Public reporting says Chamath became full-time CEO; Sina Sojoodi is cofounder and CTO.
Funding and booking numbers should not substitute for architecture evidence. Founder-reported bookings and forward targets are unaudited and are not the same thing as recognized revenue.
| Offering | Buyer | What 8090 sells | Public pricing |
|---|---|---|---|
| Software Factory | Product, design, engineering, QA, and consulting teams | A shared AI-native SDLC control plane; customer teams and their agents use it | $200 per user/month, model tokens separate; contact sales above 50 seats |
| 8090 Enterprise | Enterprises that want a finished application or modernization outcome | Discovery, custom design/build, hosting, security, maintenance, support, and operation | Starts at $1 million/year, with “Build New” and “Modernize” paths |
Source: pricing, Software Factory, and Enterprise.
This split is important for an interview. A good design answer should cover both a reusable platform and the messy last mile of enterprise delivery: identity, migration, integrations, data quality, support, SLOs, audit, incident response, and change management.
Software Factory is best understood as a multi-user, multi-agent orchestration and governance layer over the software lifecycle, not mainly as an IDE or autocomplete model.
Its canonical flow is:
Customer knowledge and business intent
↓
Requirements
↓
Blueprints
↓
Work Orders
↓
Coding agents / human engineers
↓
Tests
↓
Feedback
↺
versioned provenance and impact linksThe product calls the connected substrate a Knowledge Graph. Public material describes links from requirements to architecture, implementation, code, tests, and feedback, with forward and backward context propagation. That phrase describes the domain behavior; it does not prove that 8090 uses a graph database internally.
Requirements capture business intent: product overview, goals, personas, success measures, feature requirements, stable identifiers, user stories, and testable acceptance criteria. Relevant capabilities include collaborative editing, comments and mentions, agent assistance, version history and diffs, imports from Markdown or Word, and exports to common document formats.
The architectural implication is that a requirement is not just an unstructured page. Identity, version, approval state, authorship, referenced evidence, structured children, and outbound traceability all matter.
Source: Requirements docs.
Blueprints translate intent into engineering decisions. Public docs use C4-like Container, Component, and Feature views, plus structured blocks for components, models, contracts, assumptions, and architectural decisions. A blueprint can trace upward to requirements and downward to code symbols.
This moves engineering judgment earlier than code generation. An agent should not invent an architecture separately for every work order if the approved blueprint already defines boundaries, contracts, and nonfunctional constraints.
Source: Blueprint docs.
Work Orders turn requirements and blueprints into context-rich, dependency-aware execution units. Public examples include a stable ID, parent/child relationships, status, phase, owner, description, acceptance criteria, explicit out-of-scope items, file-level implementation plan, and end-to-end test coverage.
Agents can claim or receive Work Orders through MCP, read linked context, update status, and return proposals or artifacts. This is much closer to a durable workflow system than a one-shot prompt.
Source: Work Orders docs.
Marketing describes Tests as the fourth module and Feedback as the fifth. Test coverage can map requirement and acceptance-criterion IDs to end-to-end specifications. A documentation caveat is that the current public navigation has detailed module pages for Requirements, Blueprints, Work Orders, and Feedback, but not a standalone Tests module page. That may be a documentation or maturity gap, so the five-module story should not be treated as five equally exposed products.
Feedback arrives through the product or API, can be grouped into themes backed by evidence, and can create or update Work Orders. This closes a production loop rather than treating deployment as the end of the SDLC.
Source: Feedback docs.
Projects can ingest Markdown, office documents, images, audio, HTML, and video. These artifacts become searchable project knowledge and can be linked to assertions or generated work. The product recently renamed this area Knowledge Base and exposed search/read access over MCP.
The difficult design issue is not upload alone. A trustworthy implementation needs artifact hashes, parser and embedding versions, access-control propagation, provenance to source spans, reprocessing rules, deletion lineage, and a way to display stale or conflicting derived data.
The GitHub App is documented as read-only for selected repositories. Initial indexing takes roughly 5–10 minutes, and pushes trigger webhook-driven reindexing. Multi-repository projects and GitLab are supported. An opt-in drift analysis compares code with approved requirements and blueprints and can post a bounded set of findings on a pull request.
One lifecycle detail is notable: unlinking a repository removes its project link but does not necessarily delete already indexed material or uninstall the GitHub App. That is a useful interview edge case for retention, revocation, and deletion semantics.
Source: Codebase integration docs.
Software Factory supports external coding agents through MCP and
project- or user-scoped credentials. The public software-factory-plugin
gives agents a repeatable context → plan → checklist → review →
verification workflow and keeps execution state in
.sw-factory/. Public integrations or docs mention Codex,
Claude Code, Cursor, Gemini, Kiro, and Vercel-related workflows.
The product also lets a project connect external MCP servers through OAuth. Its Slack bot links a user’s Slack account to an organization/project, maps each Slack thread to an isolated agent session, acknowledges asynchronous work, and places attachments in project knowledge. Switching projects resets context.
These features imply several security boundaries:
The public model catalog spans providers such as OpenAI, Anthropic, Google, and Groq. Administrators can inspect token and cost usage by project, user, model, and agent. Version 0.47.0 added durable cloud agents that can keep working when a user’s computer closes, organization-level agent Skills, and Knowledge Base access through MCP.
Recent releases also expose subagents, parallel conversations, automations with run history, context compaction, retry/resume states, model selection, and agent traces. These are strong signals for a scheduler/control-plane interview: long jobs, fairness, cancellation, idempotency, retries, checkpoints, resource quotas, external side effects, metering, and reproducibility all become first-class.
Source: changelog and usage docs.
The product has private organization workspaces, projects, seats, billing, organization templates, and at least member/admin roles. It supports real-time coauthoring, comments, suggestions, version previews, and concurrent editing.
Public docs do not demonstrate a complete fine-grained enterprise IAM story such as SCIM, every-object authorization, or all deployment modes. Do not invent those features in an interview. Instead, identify them as requirements and design a path for SSO, role and attribute-based policy, service accounts, project isolation, delegated administration, emergency access, and auditable permission changes.
8090 has not published its actual database, queue, vector store, or workflow engine. Naming Neo4j, Postgres, Kafka, Temporal, Pinecone, Kubernetes, or another specific product as “what 8090 uses” would be speculation.
A safe logical architecture inference is:
Web / Slack / IDEs / external agents
│
API and identity plane
│
┌────────────┼─────────────┐
│ │ │
structured durable integration
artifact workflow and webhook
service control services
│ │ │
versioned sandboxed async
truth + workers / ingestion
event log model calls pipelines
│ │ │
└──── provenance / index / search ────┘
│
audit, metering, policy,
evaluation, and observabilityLikely system properties, based on product behavior rather than named technologies:
The marketing phrase “never drifts” should be interpreted as a goal. In a real distributed design, drift detection is delayed and fallible; the system needs freshness indicators, false-positive handling, reconciliation, and explicit approval.
The public delivery motion has three broad phases:
The company frames two paths:
This is forward-deployed engineering. The relevant system-design scope extends through rollout, migration, incident handling, cost, staffing, user adoption, and post-launch feedback.
All performance numbers below are company- or partner-reported unless an independent source is explicitly named.
This is the clearest public legacy-modernization case. 8090 says it is one of several contractors supporting CMS; it does not claim sole ownership, and CMS does not endorse the case study.
The problem includes approximately 18 million lines of Assembly and COBOL across long-lived claims systems such as CWF, DME, FISS, and MCS. The public workflow is:
The current CMS customer story reports more than 100,000 rules. A company-issued funding release on Business Wire reports more than 300,000. The discrepancy may reflect scope or timing, but the public record does not resolve it; neither number should be presented as independently audited.
Independent CMS budget material validates the larger ClaimsCore program and its need to extract and map poorly documented rules while replacing legacy shared systems. It does not validate 8090’s exact throughput.
System-design signals: massive static-analysis pipelines, source provenance, cross-system identity, reviewer queues, policy versioning, search, deterministic reruns, characterization tests, and incremental cutover.
8090 built a web application in front of BISSELL’s existing product-lifecycle-management system; the PLM remains the system of record. The app centralizes naming and numbering conventions that were previously inconsistent or tribal.
The workflow validates fields as a user types. Conforming submissions can be approved and created in the PLM automatically. Exceptions are routed to a human with a reason. A reviewer can approve the exception and evolve the rulebook, or return it for revision. Analytics show how the workflow performs.
The BISSELL story reports more than 7,000 parts, 81.4% automatic approval, a three-day average from draft to final, and about 50% lower process-plus-software cost for December 2025 through July 2026. These are company-reported production metrics, and some UI/rule examples are illustrative.
System-design signals: a deterministic rules engine, synchronous validation, exception workflows, admin-authored policy changes, dual-write/integration consistency with a legacy system of record, audit history, and feedback-driven improvement.
8090 describes replacing part of an $8–10 million/year pay-per-catch vendor workflow with a deterministic prefilter. The system handles more than 10,000 claims/day and reportedly reduced claims sent to the vendor by more than 80%, with about $21 million of four-year savings and one-year payback.
System-design signals: explainable rules, monetary correctness, decimal arithmetic, policy versioning, false-positive cost, shadow comparison, vendor routing, appeal/reconciliation, and careful rollout.
An anonymized case describes more than 30,000 faxes/week. The application classifies documents, extracts and validates data, creates patient/order records, routes exceptions, and preserves Medicare-audit evidence. 8090 reports more than 85% touchless automation and over 50 FTEs of saved effort.
System-design signals: fax/PDF ingestion, OCR and layout understanding, duplicate detection, patient matching, PHI isolation, confidence and abstention, reviewer queues, idempotent downstream writes, and immutable evidence.
8090 reports unifying more than five proprietary instrument platforms in six months and accelerating assay development by 25%.
System-design signals: device and laboratory integrations, heterogeneous data schemas, experiment lineage, offline/edge reliability, scientific reproducibility, and role-based workflows.
The described system combines clinical and claims context from more than six legacy systems and presents care opportunities during home visits. Public copy mentions more than 60 providers and more than 1,000 visits, with 20–40% of opportunities previously missed. A nearby dollar/compliance metric is malformed on the site and is not reliable enough to repeat as fact.
System-design signals: longitudinal identity, late-arriving claims, source freshness, explainable recommendations, clinician acceptance, offline field use, protected health information, and a feedback loop from care outcomes.
An unusually detailed engineering post describes a 14-month engagement with a mid-sized pharmaceutical company, in production since late 2025.
Key design choices include:
The post is refreshingly explicit about limitations: the evaluation covered a small document set and one therapeutic area, cross-customer comparison is unresolved, and judge drift requires ongoing calibration.
System-design signals: regulated RAG, retrieval coverage, evidence lineage, human edits as evaluation data, immutable audit, golden sets, judge drift, and safe model/version changes.
The Enterprise page also describes:
These examples reinforce that 8090 interviews may use an ordinary enterprise workflow with difficult reliability, policy, integration, and audit requirements rather than a consumer-social scale problem.
In March 2026, EY announced EY.ai PDLC powered by Software Factory, with a methodical deployment planned for tens of thousands of US consultants. The scope spans requirements, architecture, code, testing, infrastructure, and operations, with coordinated agents and human oversight. EY identifies legacy modernization/decommissioning and new product development as initial workloads.
EY reports one use case at more than 70% productivity/cost improvement, 80× delivery speed, and over 95% automated test coverage. Those figures are partner claims, not an independent benchmark.
This partnership is strong evidence that 8090 is designing for enterprise tenancy, high seat counts, repeatable methodology, model/tool openness, portfolio-level modernization, and reviewable agent work.
The two user-supplied repositories are recruiting and evaluation artifacts. They should not be presented as customer production systems. They are nevertheless unusually useful evidence about what 8090 considers a good engineering workflow.
top-coder-challenge: reconstruct a black-box reimbursement
policyThe top-coder-challenge
default branch is a tiny one-commit challenge package from June 2025,
not an application. It contains a PRD, five stakeholder interviews,
1,000 labeled cases, 5,000 unlabeled cases, a shell contract, and
evaluation/result scripts.
The fictional ACME scenario asks a candidate to reproduce a legacy travel-reimbursement function from three inputs:
The candidate reads contradictory stakeholder recollections, observes
historical inputs/outputs, writes a language-neutral
run.sh, evaluates against public labels, predicts the
withheld labels, and submits a separate repository plus a positional
result file.
The PRD explicitly says to preserve behavior “warts and all” before intentionally changing policy. That is classic golden-master modernization: establish parity, expose quirks, then migrate safely.
.49/.99 endings, high and low miles/day,
multifactor combinations, then a broad sample.The evaluator itself contains production-relevant defects:
< comparisons;1\n2
becomes 12;These defects are not evidence that 8090’s production platform works this way. They are excellent interview traps: an AI-generated solution can look convincing while violating its contract or gaming a weak evaluator.
Participant PRs reinforce that point. Several explicitly used Codex,
Jules, Cursor, or other agents. One claimed perfect verification even
though its run.sh did not satisfy the three-argument
contract. None of the participant PRs was merged.
What this challenge tests: requirement skepticism, behavior characterization, leakage/overfit control, reproducible evaluation, versioned financial policy, exact arithmetic, shadow comparison, and the willingness to prove agent claims with executable tests.
mib-doc-challenge: hostile, air-gapped document
decisionsThe mib-doc-challenge
is explicitly a July–August 2026 hiring challenge. All cases and
identities are synthetic. The task is to process multi-page PDFs into a
strict 12-field JSONL record and one of APPROVED,
DENIED, or NEEDS_REVIEW.
Documents mix scans, digital forms, sponsor letters, registry extracts, biometric slips, receipts, stamps, and notes. Cases include rotation, blur, low contrast, contradictions, duplicated and cross-applicant pages, rescinded decisions, hidden text, fake answer keys, malicious instructions, and fields that are intentionally unrecoverable.
The field manual establishes an explicit evidence hierarchy. Visible adjudicator notes outrank visible intake forms, biometric slips, sponsor attestations, registry extracts, and finally machine-readable text. White-on-white text, off-crop content, and instruction-shaped artifacts are untrusted.
/tmp, PID and image-size limits;The runner treats entrant containers as untrusted and advises using disposable isolated hosts or VMs. This is much more production-like than the older reimbursement harness.
The 150-point score weights adjudication most heavily, then extraction, then confidence calibration. Falsely approving a denied case is penalized much more severely than sending a decidable case to human review. Confidence uses a Brier-style calibration score. Private ranking also considers catastrophic false approvals, difficult-document slices, runtime, reproducibility, memo quality, and manual code review.
When labels depended on evidence that the PDF did not contain,
organizers explicitly advised choosing NEEDS_REVIEW, even
if guessing could improve a public score. That is powerful evidence that
honest uncertainty is a product behavior, not a failure.
A public orphan branch at commit 79d9278
labels itself an internal trial by 8090 staff. It is experimental branch
history, not default-branch production code. Its 18 Claude-authored
commits reveal a likely working style:
PRD + field policy + schema
↓
adversarial fixtures and deterministic evaluator
↓
offline parsing / visible-text checks / OCR
↓
source-ranked evidence and conflict representation
↓
deterministic policy cascade
↓
random-forest posterior + expected-value decision
↓
out-of-fold confidence calibration
↓
Docker + CI + full-batch validation + error bucketsSpecific ideas worth retaining:
CaseBelief/evidence representation separate from
output fallback guesses;The trial also contains a packaging mismatch: its Dockerfile does not copy a sibling model directory that runtime code expects. A clean image should therefore fall back from the trained model, while the CI fixtures do not assert that artifacts are present. This is a perfect example of why the canonical built artifact, not a local development run or agent summary, must be tested.
What this challenge tests: provenance-aware document processing, prompt-injection defense, uncertainty and abstention, deterministic replay, risk-weighted evaluation, air-gapped deployment, batch resilience, artifact integrity, and end-to-end rather than component-only quality.
| Factory concept | Public challenge evidence |
|---|---|
| Requirements | PRDs, stakeholder interviews, policy/field manuals, explicit business-loss asymmetry |
| Blueprints | schemas, dataset specifications, evaluator contracts, Docker constraints, architecture notes |
| Work Orders | small task-scoped commits/PRs, plans, checklists, implementation states |
| Tests | characterization cases, adversarial fixtures, CI, hidden holdouts, per-slice metrics |
| Feedback | error buckets, confusion matrices, code review, calibration, full-run artifacts, manual memo review |
This mapping is an inference, but a strong one: both challenges operationalize the structured, traceable, evaluation-heavy workflow the company markets.
The 8090 GitHub organization contains additional signals. Public code is not a complete map of proprietary production systems.
software-factory-plugin:
vendor-neutral agent skills for context, plan, checklist, review, and
verification, optionally connected to Software Factory over MCP.software-factory-harness:
an older, minimal representation of feature requirements → blueprints →
Work Orders → external agents. Older terms such as Refinery, Foundry,
and Planner show product evolution.esrd-cy212-pricer:
reverse engineering of a historical CMS ESRD COBOL pricer into
SME-readable Gherkin, dependency views, traceability matrices,
policy-version/gap logs, and regression baselines. It does not claim to
reproduce every surrounding runtime/JCL integration.medicaid-claims-data-public:
a Python/DuckDB batch analytics pipeline over a large public claims
corpus, with enrichment, statistical/temporal/network/domain rules,
multiple ML models, composite scores, holdouts, calibration,
data-quality weighting, and investigator queues. The repository
correctly warns that outliers are investigation leads, not adjudicated
fraud.delphiBank-legacy-poc: a toy/forked Pascal/Delphi
application, likely useful for legacy indexing tests; not a customer
case.Empty repositories and archived forks of third-party projects should not be treated as proprietary products.
The public “Factory Production System” describes four layers:
It also describes line operators who own end-to-end quality/yield, factory operators who improve shared throughput and guardrails, and field operators who convert customer signal into resolved work. Several capabilities are experimental, so this is as much an organizational aspiration as a mature product contract.
Current roles are on-site in Redwood City or Toronto:
Values visible in hiring material include systems thinking, end-to-end ownership, engineering excellence, curiosity, bias for action, agency, honesty, and direct communication.
Interview implication: narrate the customer outcome and operating model, not only infrastructure. Be ready to own a system after go-live.
8090 targets regulated industries, but public marketing should not be confused with a certification inventory.
A strong enterprise design should therefore make these explicit requirements rather than assumptions:
Chamath’s recent X posts reinforce this direction. He argues for model-independent harnesses, portable context, private/VPC or on-prem execution, and an immutable chain from prompt and context through model, tool calls, permissions, human approval, and final action. Treat those posts as strategic opinion, not a shipped feature guarantee: model/harness post, AI-risk audit post, and post-launch ownership.
A changelog is often a better interview guide than a marketing page. Recent fixes and additions indicate real problem classes:
These are likely high-value follow-up areas in an interview because the company has publicly encountered or prioritized them.
These are inferences, not a confirmed roadmap.
| Likely project family | Why it is plausible | Core design pressure |
|---|---|---|
| Unified provenance/versioning service | Central to Requirements, Blueprints, Work Orders, code, tests, feedback, and alignment writing | graph identity, versioned edges, consistency, impact queries, audit |
| Durable agent/automation control plane | Durable cloud agents, subagents, Skills, MCP, Slack, model catalog, metering | leases, checkpoints, retries, idempotency, cancellation, quotas, side effects |
| Multi-repo legacy intelligence | CMS, EY, GitHub/GitLab, drift bot, language expansion | incremental indexing, symbol identity, authorization, static analysis, stale results |
| Real-time structured collaboration | coediting, comments, suggestions, versions, agent mentions | CRDT/OT choices, approval authority, reconnect, concurrent human/agent edits |
| Regulated document intelligence | DME faxes, MIB challenge, pharma authoring | OCR trust, citations, PHI, abstention, reviewer queues, replay |
| Rules and policy modernization | CMS, BISSELL, reimbursement challenge, claims filtering | effective dates, determinism, parity, explainability, controlled change |
| AI evaluation and evidence platform | public challenges, medical-quality framework, agent review | golden sets, slice metrics, calibration, leakage, artifact integrity, drift |
| Enterprise integration/security plane | GitHub, GitLab, Slack, MCP/OAuth, managed delivery | confused deputy, token scopes, revocation, residency, tenant isolation |
| Claims/FWA analytics | insurer case and public Medicaid pipeline | huge batch jobs, data quality, investigator workflow, false-positive control |
| Sovereign/private deployment | regulated customers and Chamath’s public position | air gap, model artifacts, updates, telemetry, supportability, cost |
A candidate should repeatedly demonstrate these habits:
NEEDS_REVIEW as
normal product states.The local notes argue against memorized boxes-and-arrows. Both mocks below force the full RESHADED (local Grokking summary) sequence:
The interview-trap summary (local Grokking summary) also drove the staged changes. A strong candidate must adapt when scale, latency, failure, or compliance assumptions change; identify what a queue/cache/database introduces; and explain degradation rather than saying “we retry.”
You are designing Forge, a multi-tenant enterprise application used by product managers, architects, engineers, QA, and AI coding agents.
A team begins with uploaded customer knowledge and a structured Requirements document. It creates architecture Blueprints, decomposes them into dependency-aware Work Orders, sends Work Orders to coding agents, connects changes to repository symbols and tests, and ingests production feedback. Every material claim should be traceable upward to business intent and downward to code/test evidence. When any artifact changes, users need impact and drift findings without silently rewriting approved truth.
Agents may run for minutes or hours, call different model providers and tools, spawn subagents, and continue after the user’s laptop closes. Humans must be able to pause, cancel, resume, review, reject, or approve their proposals. The product must show who or what made each change, with which context, model, tool, permission, and result.
Design the complete application, including:
Out of scope for the first design: training foundation models, implementing every language parser, and building a general-purpose IDE. Define interfaces to those systems.
These are interview constraints, not 8090 production figures. Challenge them if they are internally inconsistent.
Design Forge end to end. Start by clarifying scope and invariants, then estimate load and storage. Show the core data model and APIs before the high-level architecture. Go deep on (a) the versioned provenance/impact model and (b) durable, multi-model agent execution. Explain consistency, multi-tenancy, failure handling, security, observability, evaluation, and rollout.
The prompt includes a larger follow-up bank for study, but a realistic 30-minute mock uses this route:
| Time | Interview activity |
|---|---|
| 0:00–4:00 | clarify actors, authoritative states, scope, and the cost of wrong or stale output |
| 4:00–7:00 | estimate writes, graph growth/fan-out, index traffic, and concurrent agent work |
| 7:00–13:00 | give the data model, core APIs, and control-plane/data-plane architecture |
| 13:00–21:00 | deep dive on immutable artifact versions, typed provenance, impact computation, and publication consistency |
| 21:00–27:00 | answer Reveal 1 plus either Reveal 4 or Reveal 5; revise the design under failure |
| 27:00–30:00 | cover SLOs, metrics, rollout, principal trade-offs, and a crisp recap |
Reveals not selected in this route are an extension/study bank, not questions an interviewer would fit into the same 30 minutes.
Do not volunteer all of this as the interviewer. A strong candidate should discover and prioritize it.
The candidate need not match one number, but should calculate rather than wave at “large scale.” Useful calculations include:
A strong answer gives concrete keys and invariants, not necessarily these exact table names:
Organization, Project,
Principal, RoleBinding,
IntegrationGrant;Artifact as stable logical identity and
ArtifactVersion as immutable content/metadata;Publication or release snapshot that pins a mutually
reviewed set of versions;ProvenanceEdge with source version/span, target
version/span, edge type, asserter, evidence, confidence, policy,
timestamps, and validity/supersession state;WorkOrder, dependency edge, lease/assignee, state
version, required approvals, acceptance links;Repo, Commit, FileVersion,
SymbolVersion, and index generation;AgentRun, attempt, checkpoint, tool call, model
request, side-effect receipt, budget reservation, cancellation epoch,
and final attestation;Suggestion, ApprovalDecision,
TestCase, TestRun, DriftFinding,
FeedbackItem;AuditEvent plus materialized current
views.Probe whether the candidate puts tenant_id into every
relevant partition/key and authorization path rather than relying only
on an API filter.
Require request/response shape and idempotency behavior for at least these flows:
POST /projects/{project_id}/artifacts/{artifact_id}/versions
If-Match: <base-version-etag>
Idempotency-Key: <uuid>
POST /projects/{project_id}/publications
POST /work-orders/{id}:claim
POST /work-orders/{id}/agent-runs
POST /agent-runs/{id}:cancel
POST /integrations/github/webhooks
GET /artifacts/{version_id}/impact?depth=...&cursor=...
GET /releases/{id}/audit-bundleThe candidate should explain optimistic concurrency, a conflict response, pagination/snapshot semantics for graph traversal, webhook signature verification, replay protection, and why an API retry must not launch or bill a second agent run.
Give these one at a time after the initial design. The candidate should revise the architecture, not merely append boxes.
REQ-42 v7 is published. Forty dependent Work Orders are
running, and three agents have already opened pull requests. A product
manager publishes v8 with a changed safety constraint.
Ask:
A human edits an acceptance criterion while an agent proposes a structured replacement and another reviewer bulk-accepts suggestions after reconnecting from an old browser tab.
Ask:
GitHub sends the same push three times, then delivers an older event after a newer one. Mid-index, the organization revokes the installation. A force-push removes commits that existing provenance edges reference.
Ask:
An agent times out after calling an external ticketing tool. The tool may have created the ticket, but the response was lost. The worker lease expires and another worker retries. Meanwhile the selected LLM provider is down in one region.
Ask:
An EU bank demands a dedicated VPC, tenant-managed keys, no raw code in central logs, provider zero retention, and deletion of one repository—including embeddings and every agent context derived from it—while retaining legally required approval evidence.
Ask:
One control-plane region fails while 2,000 jobs are active. A large customer submits 20,000 migration jobs and threatens to starve interactive agents for every other tenant.
Ask:
A strong design usually separates:
It should state that graph traversal/search can be eventually consistent while publication, permission, approval, budget reservation, and lease ownership need stronger invariants. It should also define a rollout: read-only indexing, advisory drift, selected projects, measured precision/recall and review burden, then carefully gated automation.
| Area | Points | Full-credit signal |
|---|---|---|
| Requirements and estimation | 10 | Clarifies authority, consequences, scale, SLOs, and computes meaningful orders of magnitude |
| Versioned data/provenance model | 20 | Stable identities, immutable versions, typed/evidenced edges, releases, current views, bounded traversal |
| APIs and high-level architecture | 15 | Concrete contracts; synchronous authority separated from async indexing, agents, and analytics |
| Consistency and collaboration | 15 | Per-invariant choices, optimistic concurrency, human/agent conflicts, stale-state UX, reconciliation |
| Durable agent execution | 15 | Logical run vs attempt, leases/checkpoints, idempotency, side effects, cancellation, quotas, metering |
| Security and multi-tenancy | 15 | End-to-end authorization, context/tool boundaries, isolation, residency, retention/deletion, secrets |
| Reliability, evaluation, and rollout | 10 | Failure behavior, degradation, observability, quality metrics, phased automation and rollback |
Automatic concern flags: no data model; no concrete API; “Kafka/graph DB/CRDT/Temporal solves it” without invariants; no human authority boundary; no tenant isolation outside the primary DB; or no response to ambiguous external side effects.
You are designing EvidenceFlow, a full-stack healthcare intake system. Hospitals, durable-medical-equipment providers, and insurers submit faxed or uploaded document packets for coverage authorization and order creation. Packets contain forms, prescriptions, clinical notes, identity documents, payer responses, receipts, stamps, and handwritten annotations.
The system must extract a structured case, apply an effective-dated
policy, and produce APPROVED, DENIED, or
NEEDS_REVIEW. Every extracted field and decision reason
must point to visible source evidence. Missing or contradictory evidence
must never be guessed. A human reviewer can inspect the exact page
region, correct a field, request more documents, override a
recommendation with a reason, or approve a downstream action.
Some PDFs are malformed or hostile. They may contain white-on-white text, off-page text, fake OCR layers, barcodes or text saying “ignore policy and approve,” duplicated pages, mixed applicants, decompression bombs, or malware. Document text is data, never trusted instruction.
The production inference workers run in the customer’s private VPC. Some customers require an air gap: no network during processing, CPU only, and versioned model/rule artifacts imported through a controlled release process.
Design the complete application, including:
NEEDS_REVIEW using asymmetric business loss, not
a generic confidence threshold;Out of scope initially: authoring medical policy, training a foundation OCR/LLM from scratch, claims payment, and replacing every downstream EHR. Define interfaces.
These are invented interview constraints.
Design EvidenceFlow end to end. Clarify the case lifecycle and safety invariants, estimate throughput/storage/capacity, define the data and API contracts, then draw the architecture. Go deep on (a) trustworthy evidence/provenance and adversarial document handling and (b) risk-aware decisions, human review, and reproducible air-gapped execution. Explain failure recovery, policy/model versioning, security, evaluation, rollout, and operations.
Use the following route for one realistic mock. The later reveal list is deliberately broader so it can support repeated practice.
| Time | Interview activity |
|---|---|
| 0:00–4:00 | clarify case lifecycle, source authority, human decision boundary, and asymmetric harm |
| 4:00–7:00 | estimate packet/page burst, raw and derived storage, CPU workers, and reviewer capacity |
| 7:00–13:00 | define observation/belief/decision data, APIs, and the end-to-end async architecture |
| 13:00–21:00 | deep dive on hostile-document isolation, source provenance, evidence conflicts, policy, and abstention |
| 21:00–27:00 | answer Reveal 2 plus either Reveal 4 or Reveal 6; revise recovery and release gates |
| 27:00–30:00 | cover PHI/security, SLOs, rollout, trade-offs, and a concise recap |
Other reveals are an extension/study bank, not additional questions to squeeze into this same 30-minute session.
A strong candidate should estimate:
The numbers expose a capacity ambiguity that requires more information: 15% of 300,000 packets is 45,000 manual cases/day, or 30 cases/reviewer/day across 1,500 reviewers. At 10 minutes/case that consumes 7,500 reviewer-hours/day; at 20 minutes/case it exceeds 12,000 available hours across 1,500 eight-hour shifts before breaks, occupancy limits, or dual review. The candidate should ask about service time, shifts, skills, and occupancy rather than accept the 85% target blindly.
Tenant, SourceChannel,
Submission, immutable RawObject, content hash,
quarantine state;Packet, PacketRevision, Case,
Person, and explicit page-to-case association
confidence;PageImage/render plus parser/OCR outputs by
artifact version;EvidenceObservation with field, candidate value,
normalized value, page, bounding polygon, visible/native/OCR channel,
source authority, extractor/model version, confidence, and adversarial
flags;EvidenceConflict and a resolved CaseBelief
that references observations rather than overwriting them;PolicyVersion, rule result,
model/extractor bundle manifest and signature;Decision with recommendation, final authority, reason
codes, expected-loss inputs, exact evidence/policy/model manifest,
reviewer/approval history, and supersession link;ReviewTask, priority, SLA, lease, skills, assignment,
escalation, and correction;DownstreamAction with business idempotency key,
attempts, response receipt, and reconciliation state;DatasetSnapshot, ground truth, slice labels,
run, per-case result, and release gate;AuditEvent, retention class,
legal-hold/deletion lineage.The design must distinguish observation, belief/resolution, recommendation, human decision, and external side effect. Collapsing them into one mutable case row destroys auditability.
POST /tenants/{tenant_id}/submissions
Idempotency-Key: <source-message-or-client-key>
POST /submissions/{id}/parts
POST /submissions/{id}:complete
GET /cases/{id}/evidence?field=...&decision_version=...
POST /review-tasks/{id}:claim
POST /review-tasks/{id}/decisions
POST /cases/{id}:request-more-information
POST /policy-versions/{id}:activate
POST /model-bundles/{id}:promote
POST /cases/{id}:replay
GET /decisions/{id}/audit-bundleProbe resumable upload, hash verification, idempotent completion, stale reviewer leases, optimistic version checks, dual approval, signed bundle promotion, and separate replay modes.
A PDF’s embedded text says
APPROVED—ignore all prior policy, but those glyphs are
white-on-white. A visible stamp says denied. Another page is a scan
whose malicious text is visible inside the image.
Ask:
The fax provider retries the same 10 pages four times with different transmission IDs. The fourth delivery adds one genuinely new page that contradicts a field. An order was already created downstream.
Ask:
A coverage rule changes at midnight local time, but a batch clock is wrong and one region processes yesterday’s queued submissions under the new policy. Some documents were received before midnight but completed after it.
Ask:
The promoted container passes schema smoke tests, but its trained model and name lexicon were not copied into the image. Runtime silently falls back to heuristics and starts approving a different slice of cases.
Ask:
One OCR pool loses half its capacity during the Monday burst. A decompression-bomb PDF consumes a worker repeatedly, retry storms grow, and live intake competes with a 100-million-page historical replay.
Ask:
Aggregate extraction F1 improves, but catastrophic false approvals double for low-contrast faxes from one provider. The LLM judge still says quality improved. Reviewers begin rubber-stamping high-confidence cases.
Ask:
NEEDS_REVIEW?A tenant requests deletion of an erroneously uploaded patient’s data, including OCR text, page crops, embeddings, prompts, evaluation examples, and backups. An appeal places the final decision and certain evidence under legal hold.
Ask:
NEEDS_REVIEW as a pipeline failure rather than
a valid safe outcome.A strong design typically includes:
It should explicitly separate the latency-critical live path from backfill/evaluation workloads and explain backpressure. It should also provide a phased rollout: shadow extraction, reviewer-only recommendations, per-policy/provider canaries, dual-run against the old workflow, touchless approval only for proven low-risk slices, rapid fallback to review, and replay/rollback.
| Area | Points | Full-credit signal |
|---|---|---|
| Requirements, risk, and estimation | 15 | Defines lifecycle/invariants, quantifies burst/storage/CPU/reviewer load, recognizes asymmetric harm |
| Evidence and versioned data model | 20 | Immutable raw/source lineage, observations vs beliefs vs decisions, conflicts, policy/model manifests |
| APIs and end-to-end architecture | 15 | Concrete contracts; durable ingress; bounded async stages; reviewer and downstream paths included |
| Adversarial safety and decision design | 15 | PDF sandbox, visible/untrusted channels, deterministic policy, calibrated abstention, expected loss |
| Reliability and air-gapped reproducibility | 15 | Idempotency, checkpoints, poison isolation, signed complete artifacts, failure-closed startup, replay/rollback |
| Human workflow, integrations, and consistency | 10 | Leases, dual approval, amendments, optimistic concurrency, business idempotency, reconciliation |
| Security, evaluation, and rollout | 10 | PHI/tenant isolation, retention/deletion, per-slice gates, hidden holdouts, phased safe automation |
Automatic concern flags: no raw-document threat boundary; no
source-level provenance; one mutable case row; no
NEEDS_REVIEW; no policy/model version in a decision;
aggregate accuracy only; no reviewer capacity calculation; or no
response to an ambiguous downstream write.
For either mock, the most 8090-aligned opening is:
“Before choosing components, I want to identify the business authority, the cost of a wrong outcome, the versioned source of truth, the human approval boundary, and what must remain reproducible after the model or code changes.”
Then use the RESHADED sequence, show concrete schemas and APIs, and keep returning to four questions:
Those questions connect the company’s platform, customer work, public challenges, engineering writing, and operating model more strongly than any guessed database brand.
mib-doc-challengemib-doc-challenge
staff trial committop-coder-challengesoftware-factory-pluginesrd-cy212-pricermedicaid-claims-data-public