Crate metrale_speculative 

2026-09-25: Metrale Engine speculative-decoding policy: the MTP gate, the adaptive and DFlash rungs, the n-gram proposer, draft-capacity clamps, per-run speculation counters and the scheduler snapshot.

Owner: speculative. Invariants: none beyond the types.

Modules

adaptive_rung

2026-09-25: Adaptive MTP draft count for batch widths 9..=16: one or two drafts, chosen from the observed accept statistics, with the static ladder (metrale_model_layers::speculative::mtp_ladder_drafts) as the floor.

dflash_rung

2026-09-25: The DFlash gamma resolver: the per-step verify width K of a block-diffusion drafter, chosen by concurrency and, for a single stream, by the first-draft accept rate.

mtp_gate

2026-09-25: The MTP runtime gate: it chooses, step by step, between the MTP verify step and plain decode by the delivered throughput (emitted tokens per second of step wall time) each mode measured over windows of WINDOW_STEPS steps.

ngram

2026-09-25: Prompt-lookup draft proposer: finds the longest suffix of the token history (prompt and generated tokens) that occurred earlier, and proposes the token that followed it. Host code only.

prompt_lookup

2026-10-02: Prompt-lookup decoding: copy proposals from a sequence’s own history (prompt and generated tokens). An index maps every n-gram of the history to the position just after its latest occurrence, so a proposal costs one hash lookup however long the history is. The proposal is the continuation of that occurrence, up to a window that doubles after a fully accepted copy and halves after a broken one. Host code only.

shared_lookup

2026-10-04: Cross-request token cache for prompt lookup. A bounded index over recent requests’ prompt and output tokens, split into scopes keyed by (model, tokenizer, LoRA adapter, tenant). A sequence whose own history has no match may copy the continuation of the same n-gram from what earlier requests of its scope held. Host memory only. The scheduler thread owns the cache, so there is no lock.

snapshot

2026-09-25: The scheduler’s observability snapshot: one Copy struct the scheduler publishes each loop tick into the run’s SnapshotCell, which the TUI reads.

spec_capacity

2026-09-25: Draft-count limits for speculative dispatch: the verify-slot capacity of every active sequence, and the serial path’s verify arm.

spec_cost

2026-10-04: The measured speculative-cost table (--spec-cost-model measured): what a verify step and a draft step cost, in wall time and GPU-rail energy, at each batch width and draft depth, on one box class for one recipe; and the drafter’s confidence→acceptance calibration. met bench spec-cost measures both and writes them; the serve loads them and plans draft depth from them. Pure: parsing and arithmetic only, no I/O.

spec_stats

2026-09-25: Per-run speculation counters: draft acceptance and per-position draft matches for the K=2, K=3 and K=4 verify arms, the B1 low-margin gauge, decode timing, and one-shot log latches. Nothing here changes generation.