A Lead Is Not a Verdict

The first rule is simple: a risk score is only a lead for an investigator. It must never deny payment or declare fraud by itself. A human reviews the source claims and decides what happens next.

I need to know the batch deadline, how corrected claims arrive, how many cases investigators can handle, and which mistakes cost more: missed abuse or a needless review. For every run, I save the exact source files, code, rules, model, and settings. That saved version list lets us reproduce any score later.

Data stewards load claim and provider data. Analysts and investigators review risk leads. Supervisors manage the limited case queue. Rule owners, compliance reviewers, operators, and auditors govern the system. Version one checks data quality, runs several detector types, groups related alerts into cases, assigns those cases, and exports the exact evidence behind a score. It learns only from cases that were actually reviewed and decided.

For this mock, a 250-million-row batch must publish within 36 hours after all inputs arrive. The case list should load in under one second for 95 out of 100 requests; an evidence page gets three seconds. We must reproduce a score for seven years. We are not blocking claims in real time, deciding criminal fraud, contacting providers, or building one identity graph across every state.

At perfect utilization one worker needs 50,000 seconds, or 13.9 hours, to scan all rows—so throughput math alone says one worker fits 36 hours. With 50% effective duty cycle and 50% failure/headroom, the one-pass equivalent rounds up to two workers. That deceptively small result ignores joins, multiple detector passes, shuffle, skew, retries, and roughly four feature facts per row. In practice shuffle partitions number in the hundreds for parallel stages and skew isolation, not because raw scan bandwidth alone requires them.

The bigger constraint is decisions: 1% produces 2.5 million raw hits, while investigators can close only 10,000 cases/day. We need grouping, suppression, calibrated risk, diversity/coverage constraints, and a fixed intake budget. Optimizing detector recall without modeling human service capacity makes the queue unstable.

I need four main records. A BatchManifest is the saved list of exact input files, row counts, code, rules, model, and settings used for one run. A ScoreReceipt explains one score and points back to its claim rows and feature values. A Lead groups related alerts into one case. A Disposition records what an investigator decided.

We never overwrite a published score. Corrected claims create a newer batch for future use while the old result remains explainable. Dates on feature records prevent a model from using information that arrived after the period being scored. APIs publish batches, search leads, open evidence, assign work, save decisions, and export a replay package.

I would draw five parts: file intake, batch processing, detectors, case selection, and the investigator app. Intake checks file fingerprints, schemas, dates, row counts, and completeness. Batch workers turn the files into a standard table and isolate bad rows for review. Rule, statistics, model, and network detectors all read the same frozen batch.

The combiner keeps each detector’s evidence, then groups related claim alerts into one provider or member case. Because investigators can review only a fixed number, a queue policy chooses the strongest cases while preserving required policy cases and a small exploration sample. Publishing the batch also saves an event for the case index in the same database transaction. Investigator decisions become training labels only after supervisor review.

Every reference lookup is time-aware and locked to a source file fingerprint. “latest provider file” is forbidden in a published run. The container image and dependency lockfile are identified by a file fingerprint. Rule DSL compiles to a stored plan. Feature definitions include null/imputation and window semantics. Stable ordering, fixed seeds, fixed-rule reducers, and explicit tie-breaking control graph/model nondeterminism. Hardware/library differences are either constrained by the release version list or accepted within a documented numeric tolerance.

A replay mode reads the original version list in an isolated ID group and writes a comparison report, never replacing history. We cross-check input row counts, partition hashes, feature aggregates, detector hit counts, lead counts, and a sample of per-entity receipts. An auditor can reconstruct one lead without recomputing all 250M rows because its evidence package contains source row IDs, feature operands, rule traces, and model/graph version lists.

I keep detector scores separate instead of collapsing them into an unexplained “fraud probability.” The combiner calibrates on adjudicated outcomes, but selection bias means observed labels are missing-not-at-random. We reserve a small random or stratified exploration sample, use inverse-propensity analysis where appropriate, and track label source and maturity. Unreviewed does not mean negative.

Queueing is a service-design problem: group duplicate claim hits into one coherent lead. Suppress known explained patterns. Respect open-case locks and investigator skills. Cap provider exposure. Reserve slots for high-severity policy rules and exploration. Then rank by expected recoverable value, harm, evidence strength, novelty, and age. Supervisors see why a case ranked, what was suppressed, and slice-level yield. Queue changes are policy versions, not hidden model tweaks.

The input version list had a completeness freshness marker. If it was violated, the published batch is marked impaired but remains reproducible. I ingest the correction as a new source version, compute data-history impact for affected member/time partitions, and build superseding batch B42. We do not mutate scores under open cases.

For untouched leads, queue policy withdraws or replaces them with an explicit reason. For acted-on cases, we attach a material-change alert, recomputed evidence, and required supervisor acknowledgment. Downstream financial action needs its own cross-checking. The data steward receives a completeness incident. Future publication can require two-source control totals and an explicit waiver when the source misses its freshness marker.

That is likely topology and compute skew, not a fraud epidemic. The graph view excludes or downweights known infrastructural hubs by typed edge semantics. It does not create a naive graph where any shared intermediary implies collusion. We monitor degree distributions and component growth before scoring. Heavy keys get salted partitions, two-stage aggregation, degree caps, and a dedicated skew lane.

I trip a detector-level circuit breaker when hit-rate or distribution shift exceeds bounds. Rules and other models can publish, but graph-derived leads stay shadowed until reviewed. A backfill runs on a corrected read-ready copy version. Score receipts make the change explicit. We never silently lower a threshold just to fit the deadline.

Each stage key is (program,batch,input_partition_digest,stage_version). Write-once outputs and a commit version list make retries safe. Scheduler queues have per-program quotas, stage concurrency, retry budgets, straggler detection, and separate urgent/live versus historical backfill pools. Publication uses compare-and-set on batch state plus a database-backed event queue. Investigator assignment uses leases and optimistic versioning so two users cannot dispose one case concurrently.

State programs have separate object prefixes/accounts, keys, catalogs, serving indexes, and worker identities where contracts require. Access policy combines program, role, purpose, case assignment, and break-glass. FTI/PHI never enters metrics. Monitor input completeness, row quarantine, stage critical path, skew, cost, score drift, calibration, flags and confirmed yield by slice, queue age/capacity, overrides, audit export, and privileged access. Replicate version lists/objects, rebuild serving stores from published batches, and rehearse regional restore with RPO/RTO evidence.

Start with data-quality reports. Then replay historical periods. Run rules/model/graph in shadow beside the old process. Show evidence to a small investigator cohort without changing queue rank. Canary queue policy with matched control investigators. Expand only when yield, reviewer time, subgroup fairness, recoverable value, reproducibility, and appeals stay within gates. Automatic adverse action remains out of scope.

The trade-off is detector recall versus constrained human attention and false accusation. A giant uncalibrated queue looks “sensitive” but wastes investigators and damages providers. I prefer an explicit capacity budget, diverse evidence, honest uncertainty, and exploration designed to learn about blind spots.

I’ll connect frozen batch truth, explainable detector receipts, bounded queues, and safe correction.