A regulated enterprise has 600 legacy applications, 80 million lines of code, thousands of batch jobs and interfaces, and incomplete ownership records. Design a modernization portfolio factory that discovers the estate, extracts behavior, prioritizes waves, coordinates parallel teams and coding agents, and performs safe strangler cutovers.
Modernize the Map, Not the Myth
Design a factory that discovers, prioritizes, and incrementally replaces hundreds of legacy applications while dependencies change beneath parallel human and agent teams.
Opening prompt
The interviewer gives a deliberately broad application problem. The candidate creates structure before drawing boxes.
I will not promise to “convert 80 million lines.” The product is a continuously verified map from business capabilities and production behavior to applications, data, interfaces, owners, controls, and migration state. First I need business deadlines, risk appetite, system-of-record boundaries, languages/runtime access, production telemetry, change rate, compliance evidence, and which outcomes mean success: cost retired, risk reduced, cycle time, or new capability.
My invariant is that every extracted rule or dependency has evidence, confidence, and freshness. Automated analysis proposes; accountable owners approve. Cutover is behavior- and data-contract driven, not lines-of-code driven.
Scope the contract
Actors, boundaries, correctness, latency, and what deliberately waits for a later phase.
Scope the first year: actors, requirements, NFRs, and exclusions.
Actors are portfolio executives, domain owners, application owners, architects, security/compliance, modernization teams, agent operators, release managers, and auditors. Version one inventories code/repos, runtime/jobs, databases, interfaces, incidents, cost, usage, controls, and ownership; builds a versioned dependency/capability graph; extracts candidate business rules and characterization tests; scores migration candidates; creates governed waves/work orders; tracks parity and cutovers; and reconciles actual retirement benefits.
Mock goals: initial 600-app inventory in 60 days; 95% evidence coverage of critical runtime flows as a portfolio discovery SLO; graph updates within four hours of source change; portfolio UI p95 under two seconds; no high-criticality cutover without dual-run and rollback; seven-year decision/audit history. The 95% aggregate is not a cutover waiver: every named critical contract in a wave needs an accountable owner, executable characterization, and rollback decision. Out of scope: automatic deletion, big-bang rewrite, translating every dead line, and letting an agent deploy without approval.
Portfolio leaders, domain/app owners, architects, security, modernization teams, agent operators, release managers, auditors.
Multi-source discovery, evidence graph, behavior/rule extraction, scoring/scenarios, waves/work orders, parity, strangler cutover, retirement.
Every edge/rule cited and versioned; conflicts visible; weighted parity; humans approve architecture and side effects.
600-app inventory in 60 days, 95% critical-flow coverage, update <4 h, UI p95 <2 s, reversible cutovers.
Read-only discovery, inherited ACLs, secret redaction, task-scoped agent tools, signed artifacts, tenant/VPC boundary.
Big-bang rewrite, automatic deletion/deploy, translating dead code, treating inventory or agent prose as truth.
Back-of-the-envelope math
These numbers are supplied mock constraints. Change them to see where the design bends.
Use 600 apps, 80M LOC, four analysis passes, 120 workers at 20K LOC/minute, 30 migration teams, and four apps/team/year. Calculate and tell me what the arithmetic hides.
Raw scan work is 320M line-passes. At 2.4M LOC/minute aggregate, the ideal wall time is 133 minutes, about 2.2 hours. That proves scanning is not the 60-day bottleneck. Building, resolving, and validating cross-system behavior, opaque binaries, data semantics, owners, and controls dominate. We need hundreds of partitions for language isolation and skew, but more workers do not create knowledge.
Thirty teams at four apps/year retire 120 apps/year, so a naive portfolio takes five years. Critical shared dependencies and change traffic reduce that throughput; reuse of extracted capabilities/contracts may improve it. The portfolio system must expose WIP, blocked dependencies, and actual retirement—not reward teams for declaring a rewrite “done” while the mainframe remains.
Analysis throughput versus migration throughput
Data, keys, and APIs
Names turn ambiguous boxes into durable contracts. The primary keys below are part of the answer.
Give the core entities, keys, and APIs. How does your graph avoid becoming an opinionated wiki?
An Application has stable enterprise identity separate from repository names. Every source artifact is content-addressed. A DependencyAssertion is versioned and stores source/target typed IDs, edge kind, direction, environment, evidence pointer, extractor version, confidence, observed time, and approval state. Conflicting assertions coexist; a materialized accepted edge is a view. Every primary key, graph partition, cache key, index document, and object path is prefixed by enterprise_id; content hashes support integrity and deduplication only inside that isolation boundary.
Business capabilities, owners, controls, costs, incidents, interfaces, data sets, rules, tests, and waves are first-class IDs. A graph version pins the evidence set used for a portfolio decision. APIs ingest discovery observations, query impact as-of a version, propose/approve assertions, create wave candidates, manage work orders, record parity results, and execute controlled traffic shifts. The wiki text is commentary; evidence-backed records drive governance.
Records and access paths
| Record | Primary / idempotency key | Important immutable fields | Main access path |
|---|---|---|---|
Application | (enterprise_id, application_id) | names/aliases, owner, criticality, lifecycle, environments | inventory; capability/owner views |
ArtifactVersion | (enterprise_id, artifact_type, content_digest) | source locator, ACL, captured_at, parser contract | incremental analysis by digest |
DependencyAssertion | (enterprise_id, edge_id, assertion_version) | typed endpoints, environment, evidence, extractor, confidence, observed_at, approval | impact as-of graph version; conflict review |
BehaviorRule | (enterprise_id, capability_id, rule_id, version) | Given/When/Then, evidence spans, SME state, effective time | characterization tests; policy diff |
MigrationWave | (enterprise_id, portfolio_id, wave_id, version) | apps/capabilities, dependencies, capacity, gates, decision rationale | roadmap/scenario; readiness |
ParityResult | (enterprise_id, release_id, fixture_id, comparator_version) | old/new outputs, side effects, value/risk slice, mismatch class | release gate; error buckets |
CutoverRoute | (enterprise_id, contract_id, route_version) | cohort, old/new target, percentage, effective time, rollback pointer | runtime routing; audit |
External contract
POST /v1/discovery/observations artifact digest + evidence
POST /v1/assertions/{id}:propose expected graph version
POST /v1/assertions/{id}:approve owner + decision reason
GET /v1/graph/impact?node=&as_of_graph= evidence-backed paths
POST /v1/portfolio/scenarios weights + hard constraints
POST /v1/waves pinned graph + capacity plan
POST /v1/work-orders/{id}:claim lease + agent identity
POST /v1/releases/{id}/parity-results signed harness result
POST /v1/contracts/{id}/traffic-shifts If-Match: route_version
POST /v1/applications/{id}:retire decommission evidence checklistEnd-to-end architecture
Control truth stays authoritative; expensive or probabilistic work is asynchronous, bounded, and replayable.
Walk through the architecture from discovery to retirement.
Read-only connectors snapshot repositories, build manifests, CMDB, schedulers, schema catalogs, API gateways, network flows, observability, tickets, and cost. An ingestion plane normalizes identifiers and stores immutable evidence. Sandboxed analyzers perform language-specific parsing, symbol/call/data-flow analysis, config/secrets redaction, SQL/job extraction, and runtime correlation. Results enter an assertion store plus graph projections and search.
A portfolio service combines criticality, change demand, cost, risk, testability, dependency centrality, owner readiness, and expected value—with inputs visible, not a magic score. Approved waves instantiate blueprint, work-order, test, and decision artifacts. Agents run in least-privilege sandboxes against frozen context; humans approve architecture and side effects. A cutover plane executes shadow reads, dual writes where safe, reconciliation, traffic slices, rollback, and eventually evidence-backed decommission.
A dependency is an assertion with evidence
Static possibility, runtime observation, owner confirmation, and freshness remain distinct instead of collapsing into wiki truth.
Deep dive on the dependency graph and portfolio ranking. Static analysis and network traces disagree; one app looks low-value but is a hub.
I model provenance and observation conditions rather than force consensus. Static analysis may show a possible call; runtime trace shows an observed call for a time/environment; a contract or owner approval adds authority. Edges carry confidence and last-seen time. We preserve negative evidence carefully—absence in sampled telemetry is not proof of no dependency.
Portfolio ranking is multi-criteria with hard constraints. A hub’s centrality and blast radius may make it a prerequisite, a platform extraction candidate, or something to stabilize rather than migrate first. I build dependency-respecting waves with capacity, blackout windows, shared data ownership, and rollback constraints. Executives can run scenarios with weights, but every recommendation includes sensitivity: if a small weight change flips the rank, it is not a robust decision.
| Evidence | Strength | Caveat |
|---|---|---|
| Static call/data flow | Broad coverage | May be unreachable or environment-specific. |
| Runtime trace/network flow | Observed behavior | Absence may reflect sampling/window. |
| Scheduler/file catalog | Strong batch dependency | Often outside application source. |
| Owner/SME approval | Business meaning | Can be stale tribal knowledge. |
| Contract/characterization test | Executable expectation | Only covers represented cases. |
Migrate contracts and behavior incrementally
A stable facade, weighted golden-master tests, single authority, reconciliation, cohort canaries, and retirement evidence tame cutover risk.
Deep dive on behavior extraction and strangler cutover. How do you know the replacement is equivalent?
We extract candidate rules from code, docs, tickets, production traces, and subject-matter interviews, each cited to evidence. We separate technical plumbing from policy and create Given/When/Then rules plus characterization fixtures from representative and edge-case production inputs. A golden-master harness compares outputs, side effects, latency, and accounting invariants using masked or synthetic data.
The strangler facade gives one stable contract. We begin shadow reads, then mirror traffic without effects, compare normalized outcomes, and investigate mismatches by value/risk slice. Where dual writes are necessary, one system remains authoritative and an outbox plus reconciliation detects divergence; we avoid uncontrolled bidirectional writes. Canary traffic increases by capability/cohort. Rollback shifts the route pointer and reconciles in-flight state. Retirement requires zero approved consumers, archived evidence, data retention disposition, credentials revoked, jobs stopped, and cost verified.
Failure injection I
The design changes under pressure. The candidate preserves correctness before convenience.
Two weeks before a wave, you discover a nightly mainframe file consumed by 41 applications. It never appeared in source code or network telemetry.
The wave readiness gate fails because critical-flow coverage is incomplete. I add scheduler and file-catalog evidence, quarantine the cutover, and use the file’s producer job, naming patterns, access audit, and downstream control totals to resolve consumers. The new assertion version triggers reverse-impact notifications and re-plans dependent waves.
Architecturally, discovery becomes continuous: code/CMDB alone was insufficient. I add batch scheduler, managed-file-transfer, filesystem audit, data lineage, and operator interview collectors. Graph completeness is measured per evidence family and critical capability, not one percentage. We may first wrap the file contract or extract a shared service instead of migrating the intended app. The schedule changes; the truth does not.
Failure injection II
Partial failure, stale inputs, duplication, and unknown external outcomes are normal distributed states.
An agent-generated replacement matches 99.7% of transactions but differs on 0.3%—which represents 38% of transaction value. Product wants to proceed.
I block cutover. Count-weighted parity is the wrong objective; value-, regulatory-, and risk-weighted slices are hard gates. The mismatch set becomes a first-class error bucket linked to rules, source lines, and fixtures. We determine whether the legacy behavior is required, a known bug, or undocumented policy; only an authorized decision can intentionally diverge.
The agent’s summary is not evidence. We rerun the canonical built artifact in the golden harness, add the high-value cases to a protected holdout, and require SME/legal approval for changed behavior. If business explicitly chooses a correction, we version the policy, migration rule, communication, and effective date. Otherwise the old route stays authoritative for that cohort while safe cohorts can canary separately.
Security and operations
Tenant isolation, backpressure, observability, evaluation, SLOs, recovery, and cost belong in the core design.
Cover concurrency across teams/agents, consistency, backpressure, security, observability, and DR.
Graph publication, approvals, wave membership, leases, and route changes use transactional compare-and-set. Analysis delivery is at least once and idempotent on artifact digest plus analyzer version. Work orders have dependency DAGs and exclusive leases for conflicting assets; optimistic merge and code ownership handle parallel changes. Per-language queues, tenant/domain quotas, retry budgets, poison-repo quarantine, and backfill pools prevent one COBOL monolith from starving change-driven analysis.
Connectors are read-only by default; code and secrets stay in tenant/VPC boundaries; extracted data inherits source ACLs; agent tools have task-scoped credentials, egress policy, signed artifacts, and human approval for writes. Observe discovery freshness/completeness, conflicting/stale edges, analysis lag/cost, WIP/block age, parity by value/risk slice, cutover error/latency/reconciliation, rollback time, retired cost, access anomalies, and audit gaps. Immutable evidence and event logs rebuild graph projections; cutover configs are replicated and restore/rollback is rehearsed.
Collector coverage, artifact freshness, unresolved identity, assertion conflict, graph critical-flow completeness.
Work in progress, blocked age, agent retries, review latency, dependency churn, reusable pattern yield.
Parity by value/risk, side-effect divergence, canary errors/latency, reconciliation lag, rollback time.
CAS approvals/waves/routes; idempotent digest analyses; leases on work; outbox side effects; versioned graph projections.
Read-only scoped collectors, inherited ACLs, per-domain keys/indexes, secret scanning, sandboxed agents, egress allowlist.
Actual systems/jobs/contracts retired, credentials revoked, cost removed, incident/risk change, cycle time—not code generated.
Rollout and trade-offs
A credible production answer defines how it earns trust and how it retreats safely.
Give the rollout and key portfolio trade-off.
Start with read-only discovery in one domain; validate identity and graph edges with owners; publish portfolio scenarios without committing funds; select a low-coupling pilot and one representative hard system; build characterization harnesses before replacement; shadow, dual-run, canary, and retire with explicit gates; then scale reusable analyzers/contracts across waves. Compare forecast with actual cost, risk, lead time, and decommission completion quarterly.
The trade-off is local migration speed versus portfolio coherence. A team can ship a new app quickly while duplicating a shared capability and leaving dependencies behind. I accept slower wave approval to make interfaces, ownership, evidence, and retirement measurable—then use the factory to accelerate repeated, well-understood patterns.
- Begin read-only in one domain with named owners.
- Validate application/source identity and evidence semantics.
- Publish scenario/sensitivity views before funding decisions.
- Pilot one low-coupling and one representative difficult system.
- Build characterization and value-weighted parity before replacement.
- Shadow, dual-run, canary by cohort, prove rollback.
- Retire only after zero consumers and control/retention evidence.
- Scale reusable analyzers, contracts, and migration patterns; compare forecast to outcome.
One-minute spoken recap
Practice this synthesis until it sounds conversational rather than memorized.
One-minute close.
I’ll summarize continuous discovery, evidence-backed decisions, parallel work controls, and reversible cutover.
Reference shelf
Definitions, traps, and the final checklist stay outside the timed mock.
- Assertion graph
- A graph whose edges are evidence-backed claims with confidence, time, source, and review state.
- Characterization test
- A test that captures what a legacy system currently does, including quirks.
- Cutover cohort
- A bounded customer, region, capability, or traffic slice routed to the replacement.
- Golden master
- Recorded legacy behavior used to compare a new implementation.
- Graph centrality
- A measure of structural importance; useful signal, not automatic migration priority.
- Strangler pattern
- Incrementally route capabilities around a legacy system until it can be retired.
- Traffic shadowing
- Run new logic on copied requests without creating business effects.
- Single authority
- During dual-run, one system remains canonical for a datum or side effect.
- Sensitivity analysis
- Testing whether reasonable scoring-weight changes alter a portfolio decision.
- Work-order lease
- Time-bounded exclusive claim on a task or conflicting asset.
- Retirement evidence
- Proof consumers, jobs, data duties, credentials, and cost were actually removed.
- Value-weighted parity
- Comparison weighted by monetary/regulatory risk rather than transaction count.
- Treating repository names as stable application identity.
- Building a graph with edges that lack source, confidence, or freshness.
- Assuming absence from sampled traces means no dependency.
- Ranking by a magic modernization score without hard constraints.
- Maximizing applications started instead of systems retired.
- Translating dead code because LOC is easy to count.
- Accepting 99.7% parity when the 0.3% carries most value.
- Allowing uncontrolled bidirectional dual writes.
- Letting agents deploy with broad persistent credentials.
- Testing a working tree instead of the signed built artifact.
- Retiring an app while batch/file consumers remain.
- Ignoring portfolio change during a multi-year migration.
- Clarified actors, authority, business harm, and out-of-scope.
- Did correct BOTEC and named the variable that changes the architecture.
- Defined stable IDs, versions, access paths, and idempotency keys.
- Separated authoritative state from derived indexes and model output.
- Explained consistency, retries, backpressure, and unknown outcomes.
- Revised the design after both failure injections.
- Covered tenant isolation, secrets, least privilege, deletion, and audit.
- Named golden signals, domain quality metrics, rollout gates, and rollback.