Module ttft

Module ttft 

Source
Expand description

2026-09-26: Warm and cold TTFT regression gates: one state machine, two modes.

  • Warm: every sample at a prompt length sends the same prompt, and each measured request follows a priming request with that prompt.
  • Cold: every sample gets a unique prefix tag, so no two requests share a prefix.

2026-09-27: The synthetic gates send prompt_lengths of filler; the high-ISL gates send one committed long prompt and check the server’s prompt-token count on every sample (ttft/long_prompt.rs).

Owner: bench, ttft. Invariants:

  • The verdict compares the run’s median and p90 with the stored baseline of the same box and model only; any other baseline gives an Info verdict (ttft_verdict.rs).
  • A failing run is never stored as the baseline (should_store).

Structs§

TtftGate

Enums§

Mode

Constants§

COLD_DESCRIPTOR
COLD_METADATA
HIGH_ISL_COLD_DESCRIPTOR
2026-09-27: expected_secs below are one-shot runs (owner, 2026-09-27) at the 2026-09-27 measurements: one 32k-token prefill per cold run (MoE 14 s, dense 45 s), plus a priming prefill before the warm one.
HIGH_ISL_COLD_MOE_DESCRIPTOR
HIGH_ISL_WARM_DESCRIPTOR
HIGH_ISL_WARM_MOE_DESCRIPTOR
WARM_DESCRIPTOR
WARM_METADATA