Module traits 

2026-09-25: The Model trait, the per-sequence state it works on (SequenceState), and the input and result types of its forward passes.

Owner: model-engine. Invariants: none beyond the types.

Structs

BeamReq

2026-09-26: One beam-search request for a translation model (NLLB): the per-request parameters the scheduler copies from the sequence. ModelForward::generate_beam_batch runs the search to completion.

ChunkedPrefillPageMetadata

2026-09-25: Per-sequence device copies of the paged-attention metadata for chunked prefill, kept across chunks so each chunk uploads only the block-table entries it added.

EpCommandFailed

2026-09-25: An EP worker command that was received and then failed to execute. The worker loop (serve_phases/build.rs) logs it and keeps running, since the head raises the same error for the same command and answers the client with it. Any other error from Model::ep_worker_step came from the receive, and the worker exits.

ForwardDisclosure

2026-09-28: What a record states about the forward.

MixedBatchResult

2026-09-25: Result of a batched mixed forward pass: M decode rows plus N prefill chunks.

MixedForwardResult

2026-09-25: Result of a mixed forward pass (decode rows plus one prefill chunk).

PrefillSlice

2026-09-25: One prefilling stream’s chunk. prefill_batch_chunk and mixed_forward_batch take a &mut [PrefillSlice<'_>] and run every stream’s chunk in one forward pass.

PrefixReclaim

2026-09-30: What ModelLifecycle::reclaim_prefix_blocks_for left: the free-block count a prefill of the prompt needs (target) and the count reached (free). free < target means nothing more was evictable.

PromptTokenLogprob

2026-09-25: One scored prompt position: log P(tokens[i+1] | tokens[..=i]) plus the top-k alternatives under the same distribution.

SequenceState

2026-09-25: One sequence’s state across prefill and decode.

VerifyBatchedOpts

2026-09-26: Per-call options for ModelVerify::decode_verify_batched.

Enums

FeedSource

2026-09-25: Where a fed decode row’s input token comes from.

ForwardSelect

2026-09-28: The forward a model’s decode runs.

Constants

DECODE_BATCH_LADDER

2026-09-28: The rungs padded_batch_n pads to.

Traits

Model

2026-09-26: The whole scheduler-facing model interface: Send + Sync and every supertrait below. An implementor writes one impl per supertrait and an empty impl Model.

ModelAdapters

2026-09-26: LoRA adapters: rotation, slot refs, disk swap and promotion.

ModelCircuit

2026-09-28: Forward selection; see the module header.

ModelDeviceFeed

2026-09-26: The device token feed of the asynchronous scheduler router, with the event queries, async copies and pinned host memory it uses.

ModelDraft

2026-09-26: Draft proposers: MTP and DFlash proposals, the hidden-state saves and stashes they read, and the DFlash context appends.

ModelEp

2026-09-26: The multi-rank (EP) command protocol.

ModelForward

2026-09-26: Prefill, decode and mixed forward passes, beam search, and the model-shape queries is_mla, hc_mult and kv_block_size.

ModelLifecycle

2026-09-26: The model’s lifecycle and memory: teardown, thread binding, sequence allocation and release, sequence swap, and the KV pool counts.

ModelLogits

2026-09-26: Reading a forward pass’s output: the logits buffers, device argmax and the post-norm hidden state.

ModelSsmState

2026-09-26: Recurrent (SSM) state: checkpoint and rollback around a verify, decode-rollback snapshots, and the commit of an accepted prefix on the secondary stream.

ModelStreams

2026-09-26: Streams, events and host synchronisation.

ModelVerify

2026-09-26: Speculative verify passes: the fixed-width, γ-wide, DFlash and batched verifies, and the GDN fold after a write-on-accept batch.

ModelVision

2026-09-26: Vision-encoder input for prompts with images.

Functions

bf16_to_f32

2026-09-25: BF16 to FP32: the two little-endian bytes become the upper 16 bits of an f32.

extract_bf16

2026-09-25: One position’s PromptTokenLogprob from a row of vocab BF16 logits.

logprob_of

2026-09-25: Log-softmax over f32_logits: the target token’s logprob and the top-k alternatives, highest first. k == 0 returns no alternatives. A target outside the vocab, or empty logits, gives -inf rather than a panic.

padded_batch_n

2026-09-25: The padded batch size for n live sequences: the smallest rung of the ladder that is at least n, or n itself above 128. Batched decode pads to a rung so its CUDA graphs are keyed by a few batch shapes instead of one per n.

Type Aliases

RowMask

2026-09-25: The two token ids the host would re-pick around for a row on the argmax fast path (u32::MAX = none): the closed-thinking mask.