Golden-Master Modernization

I would explicitly separate two products: a compatibility engine that reproduces observed legacy behavior, and a governed policy engine for intentional future changes. First freeze the observable contract: three inputs, decimal output, date/context and error behavior. Build a stratified golden-master corpus from history plus oracle probes, create a deterministic evaluator, shadow every request, and ledger both outcomes. Only after measured parity do we route cohorts. I would ask whether output alone or exceptions/timing are contract behavior too.

We need exact-cent output, input validation parity, idempotent API, effective-dated policy choice, explanation, appeal linkage, differential testing and safe rollback. The only observable features are days, miles, receipts and request date; employee or department stories are untestable and must not leak into the model. I want the historical sampling process, whether the oracle changed over time, and the legal meaning of retroactive correction. I exclude travel booking, probabilistic production decisions and silently rewriting past outcomes.

• Unknown behavior is an explicit coverage gap.
• A corrected policy version does not mutate historical decisions.
• Request date and policy effective date are distinct concepts.

Fifty thousand/day averages 0.58 RPS, so the 40 RPS peak and availability dominate compute sizing. Thirty million compact cases at roughly 100 bytes are 3 GB before indexes—easy to store; repeated evaluation is CPU-bound but parallel. A 99.99% target allows one mismatch per 10,000. Seeing zero errors in only 30,000 random tests gives a rough 95% upper error bound of 3/n = 0.01%, but random samples miss rare boundaries. I need millions of stratified and generated boundary cases, not one aggregate percentage.

50,000 / 86,400 ≈ 0.58 RPS average
99.99% ⇒ ≤ 1 mismatch / 10,000
Rule of three: zero failures in n ⇒ upper rate ≈ 3/n

Deduplicate exact input triples but preserve frequency metadata and time buckets. Stratify by days, miles/day, receipt ranges, decimal endings, extreme ratios, historical epoch and observed discontinuities. Detect output step changes and cluster local neighborhoods. Sample both traffic-weighted cases for business impact and uniform/boundary cases for rule discovery. While the oracle is available, actively probe just below/at/above candidate thresholds and create metamorphic pairs such as +1 mile or +1 cent. Every corpus version is immutable and records selection code, source hashes and expected oracle version.

• Training/discovery and certification sets are separated.
• Hold out entire strata and time ranges to detect overfit.
• Golden examples include errors and invalid inputs.

InputCase stores normalized integers: days, miles and receipt cents. OracleObservation is append-only because the same input may return differently across legacy epochs. GoldenExample pins corpus and oracle version. PolicyVersion is immutable after approval, with non-overlapping effective intervals enforced per policy line. Decision is keyed by tenant/request ID and stores input hash, selected policy, cents and explanation hash. A correction creates a new version; an appeal references the old decision and any adjustment is a second decision. POST requires an idempotency key and effective context.

The edge authenticates and dedupes request ID. A strict normalizer parses decimal strings to integer cents, validates units and canonicalizes the input hash. The policy router selects the one approved version whose effective interval contains the decision date, and snapshots that choice. A deterministic rules engine evaluates a compiled decision table, producing output cents and rule trace. In one transaction, the service writes Decision plus an outbox response event; retries return the same decision. During shadow, an asynchronous adapter invokes or reads the legacy outcome and writes ShadowComparison. Dashboards aggregate deltas, but the immutable ledger remains the investigation source.

• Hot path does not depend on the legacy oracle.
• Policy bundle is cached by immutable hash.
• Shadow comparison cannot change customer response.

Start with interpretable hypotheses from discontinuity plots and interviews, but certify only against held-out observations. A decision-table/rule DSL encodes tiers, caps, bonuses and interactions with explicit rounding after each stage if observations require it. Use active learning to query cases that distinguish competing hypotheses. Metamorphic tests find nonmonotonic zones but do not assume monotonicity globally. Keep an exception table only for genuinely isolated known legacy defects; a giant lookup/memorization model would fail unseen combinations and be impossible to govern.

• Measure exact match, absolute-cent delta and weighted financial delta.
• Slice by every discovered boundary and historical epoch.
• An inferred rule carries evidence cases and confidence, but production code is deterministic.

Each approved PolicyVersion has transaction time and valid effective interval. The router resolves using the business decision date, not deploy time. A correction proposal includes rationale, impacted golden strata, simulated financial delta, approvals and intended effective date. If retroactive recalculation is allowed, generate adjustment decisions referencing originals; never mutate ledger rows. Overlap constraints prevent two active versions for the same cohort/date. The golden suite has two lanes: compatibility tests expected to match legacy and policy tests expected to differ by an approved correction ID.

• Bitemporal metadata distinguishes what was believed when from when policy applies.
• Rollback routes new requests to prior bundle; already issued decisions remain traceable.
• A deliberate mismatch is labeled, budgeted and reviewed.

Strong consistency covers request dedupe, policy interval publication and decision ledger. Corpus evaluation and comparison analytics are eventual. Idempotency key maps to canonical input hash; reusing a key with different input returns 409. Decision plus outbox commits atomically, so downstream payment sees one effect despite at-least-once delivery. If policy selection or ledger is unavailable, fail closed rather than guess money. Shadow adapter failures do not fail customer requests; they create a comparison gap metric and retry with bounded age.

OracleObservation already includes observed time and environment, so we detect a multimodal output for one fingerprint. Freeze certification against the pre-change oracle version, stop active promotion, and create a new epoch. Probe a stratified panel before and after to localize affected regions. Ask policy owners whether it is an intended policy change or defect. Do not blend observations and learn an average. The shadow comparator labels legacy epoch, and routing remains on the last approved new-engine policy until a reviewed version models the intended change.

Phase zero replays frozen history. Phase one mirrors live requests asynchronously while legacy remains authoritative. Phase two makes the new engine authoritative for internal/test accounts, then low-risk cohorts selected by stable hash—not per request—to avoid inconsistent employee outcomes. Every phase has parity, financial-delta, latency, error and appeal guardrails with an automatic routing kill switch. Keep dual computation long enough to cover monthly boundaries. At full cutover, retain a read-only legacy adapter and replay capability until retention and audit requirements expire.

• Shadow observes production distribution without double-paying.
• Cohort routing and policy version are recorded in Decision.
• Rollback changes future routing; it does not delete decisions.

Pause cohort expansion because aggregate 99.99% can hide a material slice. Create a dedicated boundary stratum around those endings and inspect each arithmetic stage. This likely indicates decimal parsing, rounding order or a threshold keyed to cents. Replace any floating-point path with integer/rational arithmetic, encode rounding mode at the exact legacy stage, and add metamorphic cases ±1 cent. Backfill comparisons to calculate total financial impact. Promotion requires slice-specific zero unexplained mismatches, not only global parity.

The three numeric inputs are low sensitivity alone, but production requests may carry employee and tenant identity. Minimize the evaluator input, tokenize actor identity, encrypt ledgers with tenant keys and isolate test corpora from raw PII. Policy authors, approvers and deployers have separated roles. Golden datasets are access-controlled and exports watermarked/audited. Signed bundle hashes and append-only decisions detect tampering. The oracle adapter uses least-privilege credentials and cannot issue payment. Explanations redact internal anti-fraud thresholds when policy requires.

• No model call is needed in the money path.
• Test data preserves boundary behavior without exposing names.
• Appeal evidence has stricter retention and access.

Service SLIs: p99 latency, availability, dedupe conflicts, policy-cache miss, ledger commit and outbox lag. Correctness: exact-cent parity, absolute delta, signed financial delta, mismatch rate by stratum/epoch/policy/cohort and validation-error parity. Migration: shadow coverage, legacy adapter gap age, cohort size and rollback readiness. Product: appeals, overturn rate, investigator time and unexplained mismatch age. Every release publishes a matrix over frozen corpora; online monitors compare distributions and run synthetic boundary canaries.

• Alert on cents, not only request counts.
• Reconcile Decision outbox with downstream payment ledger.
• Retain the exact executable rule bundle for replay.

Interpretability may take more engineering than a flexible model but is required for exact money and corrections. Mirroring the legacy bug maximizes compatibility while policy cleanup improves outcomes; keeping separate compatibility and correction lanes resolves that tension. Traps are random-only sampling, leakage from history into certification, floats, assuming calendar/person inputs that do not exist, one global parity metric, mutating old decisions, and big-bang cutover.

ReimburseX learns with evidence but runs deterministically. Immutable, stratified golden corpora capture observed legacy epochs, boundaries, errors and metamorphic probes. A compiled, integer-money policy engine selects an approved effective-dated bundle and writes an immutable idempotent decision plus explanation. Shadow comparison is asynchronous and sliced by risk; deliberate corrections have their own approvals and expected mismatches. We progress from replay to shadow to stable cohorts with financial guardrails and rollback. Historical decisions remain untouched; appeals and adjustments form a traceable ledger.