Evaluate agents under adversarial daylight.

I see four authority boundaries: specification, execution, ground truth, and scoring. Entrant code is untrusted. Private inputs/labels and metric code are privileged. A score is valid only for an immutable tuple: submission image digest, dataset snapshot, runner image/architecture, metric version, resource profile, environment/seed, and completed case-result manifest.

I will clarify submission languages/artifacts, interactive versus batch agents, network/tool simulation, deterministic requirements, per-case deadline, public feedback budget, risk weights, tenant privacy, and what “reproducible” means across CPU architectures. I will never compute a flattering metric over only successful rows.

Actors are evaluation author, dataset curator, entrant/engineering team, CI service, sandbox scheduler, human reviewer, security operator, and release approver. The lifecycle is define spec → register versioned datasets/slices/metrics → upload and scan immutable image → public smoke/feedback run → quota-controlled private run → shard execution → strict output validation → deterministic aggregation → slice diagnostics/manual review → signed scorecard → promote or reject.

Functional requirements include private/public partitions, stratified slices, resource profiles, resumable shards, hidden labels, result access controls, leaderboard/project views, run comparison, artifact retention, appeals, and rerun policy. NFRs: no runtime network, P95 queue start under five minutes for smoke runs, full-run completion under two hours, deterministic score equality within an explicit tolerance, 99.9% scheduler availability, and no cross-tenant/label leakage. Foundation-model training is out of scope.

1. Inputs, labels, slice membership, metric code, and submission artifact are content-addressed and immutable for a run.
2. The entrant sees only the documented input view; labels, private slice names, other tenants, secrets, clocks, network, and host metadata are unavailable.
3. Exactly one terminal result exists per expected case ID: valid output, timeout, crash, resource breach, malformed output, or missing.
4. Every terminal category contributes to the denominator and policy-defined loss.
5. Parser behavior is strict and versioned—no whitespace concatenation, locale guessing, duplicate IDs, NaN/Inf, or silent coercion.
6. A shard retry never changes submission/dataset/runner tuple; aggregation consumes one fenced accepted attempt per case.
7. Private feedback and submission frequency are budgeted so leaderboard probing cannot become label extraction.

Two thousand submissions × 500,000 cases × 0.15 CPU-seconds is 150 million CPU-seconds/day, or 1,736 continuously busy cores. At 70% target utilization I provision around 2,480 cores before P99 and security overhead. This immediately suggests tiers: cheap smoke/public runs for every commit, full private runs only by policy or quota.

With 32 single-core shards, one full submission uses 15,625 cases/shard × .15 s ≈ 39 minutes ideal wall time; startup, skew, retries, and aggregation push the SLO higher. One billion case results/day at an 80-byte compact record is 80 GB/day, while stdout/artifacts must have strict caps and short hot retention. Datasets are mounted once per node/cache by immutable digest rather than copied per run.

EvaluationSpecVersion pins input/output schemas, resource profile, metric definitions, failure penalties, determinism tolerance, shard plan, and feedback policy. DatasetSnapshot pins objects, case IDs, private labels, and slice membership. Submission pins the scanned image digest and owner. Run pins all three plus runner/environment manifest before scheduling.

Each ShardAttempt has a lease epoch and output object hash; a committed CaseResult names case ID, terminal status, parsed value hash, duration/resource counters, and attempt. Aggregation first validates the complete expected case-ID set, then computes metric components from a canonical ordered record stream. APIs return run/score IDs, never mutable “latest,” and idempotency makes CI retries return the same logical run.

The artifact gateway authenticates tenant/team, performs resumable upload, hashes the OCI image/archive, scans malware/SBOM/policy, and stores it immutable. The run API authorizes a spec and run tier, reserves compute quota, pins the manifest, and emits a scheduler event. Dataset staging decrypts private inputs inside the isolated evaluation boundary; label objects are mounted only to the scoring service—not the sandbox.

The scheduler shards stable case IDs, leases sandbox nodes, and records heartbeats/resource usage. Each sandbox has no network, read-only root/input, minimal env, bounded CPU/memory/PIDs/scratch/output/wall time, deterministic clock/randomness contract, and a trusted output side channel. Validators parse strict framing and create one terminal record per case. The aggregator verifies completeness, reduces deterministic metric components, computes slice scorecards, and signs an attestation. Public UI sees only policy-approved feedback.

I pin image digest, runner image, kernel/runtime profile, CPU architecture class, locale/timezone, dependency artifacts, input manifest, seeds, thread limits, and metric version. I remove network, cloud metadata, host mounts, privileged syscalls/capabilities, wall-clock detail, and ambient secrets. Namespaces plus cgroups are necessary but I prefer microVMs or equally strong isolation for hostile multi-tenant code.

Determinism is defined by contract. Integer/schema outputs should match exactly. Floating metrics specify canonical decimal parsing, stable ordering, numerically stable reduction, and tolerance. If the submitted model is inherently stochastic, the spec supplies deterministic seeds and may evaluate repeated trials; the attestation records each seed. A clean-room rerun pulls only content-addressed artifacts—never a developer cache.

We store stdout/stderr separately with byte caps and secret scanning. A failed process is not rerun merely to collect stderr because the second execution could behave differently or leak information.

Phase one validates the result set: expected IDs, exactly one row each, schema version, bounded UTF-8 framing, allowed finite numbers, checksums, and no extra rows. Every case receives a terminal status. Phase two assigns policy loss: valid prediction metric, timeout penalty, crash penalty, malformed penalty, and often catastrophic-policy penalties. The denominator is the full expected set.

For the reimbursement-style example, MAE over successes alone is wrong: one exact answer plus 499,999 crashes cannot beat a complete imperfect model. Likewise inclusive tolerances must be implemented as abs(error) <= threshold; parser whitespace must never turn 1 2 into 12; declared timeout, memory, output, and PID limits must actually be enforced.

Aggregation uses stable case ordering and pairwise/Kahan-style sums where needed. Metric code has golden unit/boundary/property tests and a separate reference implementation. The scorecard displays coverage, every failure bucket, weighted loss, confidence calibration, and slices—not one scalar alone.

I freeze the leaderboard and mark affected scorecards invalid; do not silently recompute history. I issue a new metric/spec version with complete-set validation and explicit terminal penalties, run a migration audit to identify affected runs, and recompute into new signed scorecards linked to the originals. Entrants get a transparent incident note and appeal path.

Prevention adds invariants before metric reduction, golden adversarial submissions that crash/timeout/omit/duplicate/emit NaN, property tests that worsening any case cannot improve coverage-adjusted loss, and differential testing against a simple reference evaluator. Release requires full canonical container execution, not only metric-unit tests.

There is no network namespace route or DNS resolver, but I also remove clock precision, host contention signals, case filenames/order clues, and per-case feedback. The runner can randomize or batch case order from a hidden seed and equalize label-dependent scoring work because labels never enter the sandbox. Public results expose coarse aggregate/slice feedback only after minimum cohort sizes.

Private evaluation has submission quotas, cooldowns, similarity detection across near-identical submissions, and a privacy budget for feedback. Final holdout inputs/labels rotate and are inaccessible to normal developers/CI. Output bytes, files, PIDs, syscalls, memory, CPU, scratch, and wall time are bounded outside entrant code; violations become terminal results. Network or sandbox escape attempts trigger security quarantine, not merely a low score.

I monitor unusual query sequences, score deltas, timing correlation, artifacts trying to read host paths, and leaderboard overfitting. A final release decision also uses a never-before-seen confirmation set.

Not until hard-gate slices pass. Slice definitions are versioned, private where they could leak labels, and can overlap. I require minimum support or confidence intervals; tiny slices trigger qualitative review rather than noisy rank changes. The scorecard shows coverage, central metric, worst-slice loss, catastrophic count, calibration, runtime/resource distribution, and reproducibility across reruns.

Schema/document drift is itself a slice: if prose says mileage integer but data contains floats, the canonical schema and validation behavior must be explicit. We do not coerce silently. I compare against the incumbent per slice and weight business harm, not merely sample frequency. A global gain cannot average away a safety or contractual regression.

Hierarchical queues allocate tenant/team quotas, interactive smoke capacity, scheduled private evaluations, and low-priority backfills. Weighted fair scheduling with aging prevents one team from occupying the fleet; admission control reserves full-run budget before launch. Shards checkpoint only trusted result manifests; deterministic failures get finite retries, while node failures can retry with the same tuple and fenced attempt.

I monitor queue age, core utilization, sandbox startup, timeout/resource-breach rate, shard retries, result completeness, score determinism, metric drift, label-access attempts, private feedback budget, slice sample health, attestation gaps, and cost per valid evaluation. Rollout begins with internal reference submissions and shadow scoring, then public smoke runs, then small private canaries, then promotion gates. Old evaluator versions remain reproducible but cannot masquerade as current.

Glassbox Arena pins submission, dataset, runner, metric, resource, and seed into one immutable run. Hostile code executes without network or secrets in resource-bounded isolation; labels live only in the scoring plane. Every expected case gets a terminal result, so crashes and malformed output cannot improve the metric. Aggregation is deterministic, slice-aware, and signed.

Private feedback is budgeted, timing and output side channels are constrained, and promotion uses a final confirmation set plus reproducible clean-room reruns. The trade-off is rich feedback versus benchmark secrecy: I expose enough public diagnostics to improve engineering while rationing private information so teams cannot train on the test.