Expand description
2026-09-25: Fixed-stride batched-decode metadata layout, derived from the serve max_batch_size.
sizes.rs sizes scratch from it, and metrale-model-engine’s
upload_batch_metadata_fixed / _at take their offsets from it. Byte
offsets, with R = rows:
positions u32 [0, 4R)
seq_slot i32 [4R, 8R) (per-request LoRA slot)
slots i64 [8R, 16R)
seq_lens i32 [16R, 20R)
(unused [20R, 24R))
block_tbl i32 [24R, 24R + R·max_blocks·4)
Owner: gpu-runtime. Invariants:
rows() >= DECODE_META_MIN_ROWS.- The regions do not overlap, and
slots_off()is a multiple of 8.
Structs§
- Decode
Meta Layout - 2026-09-25: The decode-metadata layout for
rowsrows (see the module docs).
Constants§
- DECODE_
META_ MAX_ ROWS - 2026-09-25: The largest serve
max_batch_sizeaccepted;serve_load.rsrefuses a larger one at startup. The layout itself has no upper bound. - DECODE_
META_ MIN_ ROWS - 2026-09-25: The row floor:
rows = max(DECODE_META_MIN_ROWS, max_batch_size).