Module benchmarks 

2026-09-26: The benchmark drivers, and the text helpers they share.

Owner: bench. Invariants: none beyond the types.

Modules

agentic

2026-09-26: The agentic-webserver benchmark: N iterations of one task (write a Rust Axum ping/pong project, test it, run it, curl it, tear it down), each in a fresh sandbox. After each iteration the scorer builds and runs what the agent left and asks /ping for pong, and checks the agent’s commands against the prompt’s directions.

baseline

2026-09-26: Per-benchmark baselines: the metrics of an earlier run, stored as JSON in the artifact store’s runs/<benchmark_id>/ directory, for the benchmarks that compare a run against their own history (ttft, serve_matrix). Each file records the endpoint and model it was measured against.

bfcl

2026-09-26: BFCL v4 single-turn, full and subset draws, whole or sharded.

concurrency

2026-09-26: The concurrency sweep: for every (ISL × concurrency) cell, send conc streaming requests at once and report client TTFT, TPOT and E2E percentiles plus the batch’s aggregate output tok/s. Each next() runs one cell. Every request’s delivered evidence is kept, and a cell that is vacuous (concurrency_vacuity.rs), errored or cache-uncontrolled is left out of the gate metrics.

contamination

2026-09-26: Cross-contamination benchmark: concurrent requests must not change each other’s output. score holds the leg design and the classification, prompts the probe corpus, and driver the state machine that runs the legs.

decode_floor

2026-09-26: Decode Floor Gate: the pinned single-user decode throughput floor. Every generation knob is pinned so that two runs are comparable, and the one metric, the median server decode rate over RUNS runs, is judged against the served variant’s BENCH.toml server_decode_tok_s bound under --pull-request-gate. decode-floor is listed in gate::coverage::REQUIRED.

default_tier_boot

2026-10-01: Default-Tier Boot: a boot-only check of the serve’s memory plan. Once the serve is up it sends one short warmup wave, then reads GET /memory and reports the KV pool’s block count and whether the serve’s device footprint stays inside its --gpu-memory-utilization budget.

kat_equality

2026-09-26: KAT equality: the same sample, in different request orders, byte for byte.

media_integrity

2026-09-26: Cross-request and cross-path integrity checks, used by the vision and video benchmarks. They target machinery a single well-formed request cannot reach, whose failure is a fluent answer to the wrong input rather than an error:

mlperf_agentic

2026-09-26: MLPerf Agentic Inference: teacher-forced replay of recorded trajectories with inline scoring. The prompt for each client turn is the recorded history plus the current client message (dataset), so the model’s output is scored but never fed forward. Scoring ports the upstream inline scorer’s two rules, workflow intent-code match and coding bash-executable multiset IoU (scoring).

quick_speed

2026-09-26: Quick Speed Bench: warmup plus N timed streaming runs of one prompt, reporting the server’s decode rate, TTFT and end-to-end latency. A measurement tool, not a gate: it stores no baseline, reads no threshold, and gate::coverage::NOT_REQUIRED lists it.

scheduler_equivalence

2026-09-26: Scheduler equivalence: one BFCL draw served under the sync and the async device router (--scheduler-config), compared per sample_id and per concurrency.

serve_matrix

2026-09-26: Serve Matrix: boot every checkpoint the box can serve, run the coherence, codegen, tool-call, long-context and throughput probes on it, and score one row per round.

spec_cost

2026-10-04: Speculative step cost calibration: on the serve it is pointed at, the wall time and GPU-rail energy of ONE speculative scheduler step at each batch width n, for the draft depth k the serve runs, split into its draft (propose) and verify shares. A measurement tool for the speculative cost model, not a gate: gate::coverage::NOT_REQUIRED lists it.

ssm_poison

2026-09-26: SSM state poisoning gate: driver runs it, probe holds the script, compare judges each replay round, score decides, toolcall checks path-independence of tool calls, and report renders.

stats

2026-09-26: Shared measurement helpers: accept length, prompt synthesis, percentiles, median and formatting.

transcript

2026-09-26: What a request emitted, reduced to the parts the equality gates compare (contamination, scheduler equivalence, KAT equality, SSM poisoning).

ttft

2026-09-26: Warm and cold TTFT regression gates: one state machine, two modes.

video

2026-09-26: The video-fidelity benchmark: did a clip’s frames reach the model, in order?

vision

2026-09-26: The vision-fidelity benchmark: does the served model see the image it was sent, at the resolution the serve permits?

Functions

one_line

2026-09-26: Collapse a message onto one line (whitespace runs become one space) and bound it to 300 characters, the last an ellipsis.

unique_prefix_tag

2026-09-26: A prompt prefix tag of the form <prefix>-<pid>-<run_id>.