Module seq_state_reserve

Module seq_state_reserve 

Source
Expand description

2026-09-25: Per-sequence device state the serve must reserve, computed from the config before the model exists.

The server’s preflight_reserve adds this charge (per_sequence_reserve) before the model is built, when there is no layer list, proposer or GpuBackend, so it is a function of the config alone, like the ssm_reserve terms.

Charged, per sequence:

  • the target stack’s DSA indexer caches (Glm5NextDsaState), one per text DSA layer;
  • the draft proposer’s own state (what Glm5NextMtpHead’s alloc_state allocates), in its own field: the proposer is not a TransformerLayer and is not in the layer list.

Not charged here:

  • KDA recurrent and conv state: pool-owned (ssm_reserve::PoolPlan); meta.rs gives pool-backed linear-attention layers pool addresses instead of calling alloc_state;
  • the paged KV pool: this reserve is subtracted from the KV budget.

The result is per-rank bytes; nothing here divides by a world size.

Owner: model-arch (serve memory reserve). Invariants:

  • Any model_type other than glm5_next is charged zero.

Structs§

PerSequenceState
2026-09-25: Per-sequence, per-rank device state, split by owner.

Functions§

per_sequence_state_bytes
2026-09-25: Per-sequence owned device state for config, at a context of max_seq_len tokens.