Struct BufferSizes 

pub struct BufferSizes {

    pub hidden_states: usize,
    pub residual: usize,
    pub norm_output: usize,
    pub qkv_output: usize,
    pub attn_output: usize,
    pub gate_logits: usize,
    pub gate_logits_f32: usize,
    pub moe_router_in_f32: usize,
    pub moe_output: usize,
    pub logits: usize,
    pub ssm_qkvz: usize,
    pub ssm_ba: usize,
    pub ssm_deinterleaved: usize,
    pub ssm_gates: usize,
    pub ssm_conv_out_f32: usize,
    pub scratch: usize,
    pub expert_gate_out: usize,
    pub expert_up_out: usize,
    pub expert_down_out: usize,
    pub splitk_workspace: usize,
    pub gdn_fla_scratch: usize,
    pub ssd_scratch: usize,
    pub o_latent: usize,
    pub norm_unit_w: usize,
    pub hc_streams: usize,
    pub hc_post: usize,
    pub hc_comb: usize,
    pub hc_lowrank_scratch: usize,
    pub qsa_select_scratch: usize,
    pub token_ids: usize,
    pub ffn_act_q8: usize,
    pub ffn_act_a: usize,
    pub ffn_act_scale: usize,
    pub ffn_act_scale_kmajor: usize,
    pub ffn_gate_up_fused: usize,
    pub fp8_act: usize,
    pub moe_fp8_scratch: usize,
    pub fp8_act_scale: usize,
    pub fp8_act_scale_kmajor: usize,
    pub lora_xa: usize,
    pub lora_delta: usize,
    pub lora_hact: usize,
    pub lora_seq_slot: usize,
    pub q2_dequant_scratch: usize,
    pub q2_act_q8: usize,
    pub ssm_rowwise_w_bf16: usize,
}

2026-09-25: Byte size of each arena buffer. M is max_batch_tokens and ceil16(M) is M rounded up to a multiple of 16.

Fields

hidden_states: usize

residual: usize

norm_output: usize

qkv_output: usize

attn_output: usize

gate_logits: usize

gate_logits_f32: usize

2026-09-25: FP32 router logits [M, num_experts + zero_expert_num] for the METRALE_FP32_GATE and METRALE_FP32_ROUTING paths, which run the top-K on unrounded logits. 256 bytes for a model without experts.

moe_router_in_f32: usize

2026-09-25: FP32 MoE-input norm output [M, hidden] that the router GEMM reads under METRALE_FP32_ROUTING. 256 bytes for a model without experts.

moe_output: usize

logits: usize

ssm_qkvz: usize

ssm_ba: usize

ssm_deinterleaved: usize

ssm_gates: usize

ssm_conv_out_f32: usize

scratch: usize

expert_gate_out: usize

expert_up_out: usize

expert_down_out: usize

splitk_workspace: usize

gdn_fla_scratch: usize

2026-09-25: GDN FLA chunked-prefill scratch, W | U | S | uc | gc back to back. 0 unless the linear-attention key and value head dims are both 128, the only shape the FLA prefill kernels run.

ssd_scratch: usize

2026-09-25: Mamba-2 SSD chunked-scan scratch, dt | dA_cumsum | CB. 0 unless the model has Mamba-2 heads and a state size.

o_latent: usize

2026-09-25: Grouped O-projection latent [M, o_groups * o_lora_rank] BF16, at least 256 bytes.

norm_unit_w: usize

2026-09-25: The zero weight norm_unit_w, max(hidden, Mamba-2 d_inner) BF16 values.

hc_streams: usize

2026-09-25: Hyper-connection residual streams [M, hc_mult, hidden], FP32. 256 bytes when hc_mult == 0.

hc_post: usize

2026-09-25: Hyper-connection post weights [M, hc_mult] F32.

hc_comb: usize

2026-09-25: Hyper-connection comb matrix [M, hc_mult, hc_mult] F32.

hc_lowrank_scratch: usize

2026-09-25: Low-rank hyper-connection scratch: the larger of the decode split layout (T <= 64, FP32) and the prefill GEMM layout (slabs of at most 2048 tokens, BF16); see the sizing in from_config. 256 bytes without a low-rank hyper-connection.

qsa_select_scratch: usize

2026-09-25: QSA prefill-selection scratch for 2048-row slabs, shared by the indexer layers: qk [2048, (index_n_heads + 1) * index_head_dim] BF16, q_post [2048, index_n_heads, index_head_dim] F32, scores [2048, ceil(max_seq_len / index_compress_ratio)] F32, lists [2048, index_topk / index_compress_ratio] i32. 256 bytes without an indexer. layers/qsa_select.rs carves the same layout.

token_ids: usize

2026-09-25: Token ids [M] u32 of the current pass, read by the DeepSeek-V4 hash-routed MoE layers (tid2eid[token_id]). At least 256 bytes.

ffn_act_q8: usize

2026-09-25: Dense-FFN activation-quant scratch shared by every layer’s prefill, sized for K = max(hidden, intermediate); all 0 for a MoE model. ffn_act_q8: q8_1_mmq activations for the Q4_K MMQ path. ffn_act_a: int8 [ceil16(M), K], also NVFP4 packed and FP8 activations. ffn_act_scale: [ceil16(M), K/32] 4-byte int8 scales, also the NVFP4 and FP8 scales.

ffn_act_a: usize

ffn_act_scale: usize

ffn_act_scale_kmajor: usize

2026-09-25: [K/128, ceil16(M)] f32 copy of the FP8 activation scales, token index contiguous, for the cuBLASLt block-scaled FFN GEMM; the quantizer writes [M, K/128] into ffn_act_scale. 0 for a MoE model.

ffn_gate_up_fused: usize

2026-09-25: [ceil16(GATEUP_FUSED_MAX_M), 2 * intermediate] BF16 output of the fused dense-FFN gate+up decode GEMM; a row is [gate | up]. A buffer of its own because the fused arm serves only the decode band, and a widened expert_gate_out would carry max_batch_tokens rows. Sized for every dense model, whatever the lever says: the levers are resolved above this crate. 0 for a MoE model.

fp8_act: usize

2026-09-25: FP8 activation scratch for the prefill projections, 1 byte per element, [ceil16(M), K] for the widest K among hidden, q_heads * head_dim, Mamba-2 d_inner and the GDN value dim.

moe_fp8_scratch: usize

2026-09-25: The grouped FP8 MoE scratch slab (moe_fp8_scratch::Layout); 0 for a dense model.

fp8_act_scale: usize

2026-09-25: One f32 scale per 128 elements of fp8_act.

fp8_act_scale_kmajor: usize

2026-09-25: [K/128, ceil16(M)] f32 transpose of fp8_act_scale, token index contiguous, for the cuBLASLt block-scaled projections. Sized for every model, the SSM in_proj_qkvz arm included.

lora_xa: usize

2026-09-25: LoRA shrink output xa = x @ Aᵀ, [M, adapter_max_rank] BF16. This and the other lora_* entries are 0 when adapter_max_rank == 0.

lora_delta: usize

2026-09-25: LoRA expand output delta = xa @ Bᵀ, [M, n] BF16 with n the widest target: max(hidden, intermediate, the q projection width).

lora_hact: usize

2026-09-25: LoRA hidden-activation scratch [M, intermediate_size] BF16.

lora_seq_slot: usize

2026-09-25: LoRA adapter slot per prefill token, [M] i32. A buffer of its own, so it cannot overlap the metadata regions of scratch.

q2_dequant_scratch: usize

2026-09-25: Keep-packed Q2_0 prefill dequant scratch (METRALE_GGUF_NATIVE_Q2=1): one BF16 buffer for the largest keep-packed projection, reused by each projection’s dequant. Sized by crate::buffers::q2_dequant_scratch_bytes; 0 unless the lever is 1.

q2_act_q8: usize

2026-09-25: Keep-packed Q2_0 MMQ prefill q8_1 activation scratch (METRALE_GGUF_NATIVE_Q2_MMQ=1), shared by every keep-packed projection: each quantizes its BF16 activation here and runs the packed MMQ GEMM. 0 unless the lever is 1.

ssm_rowwise_w_bf16: usize

2026-09-25: Row-wise FP8 GDN prefill BF16-weight slab (METRALE_FP8_ROWWISE=1): every GDN layer’s BF16 in_proj_qkvz and out_proj, one slice pair per layer carved on its first prefill and kept for the arena’s life. Sized in sizes_rowwise.rs; 0 unless the lever is 1.

Implementations

impl BufferSizes

pub fn total_bytes(&self) -> usize

2026-09-25: The sum of the sizes, which preflight reserves for the arena. o_latent and norm_unit_w are not in it.

impl BufferSizes

pub fn from_config( config: &ModelConfig, max_batch_tokens: usize, max_seq_len: usize, kv_block_size: usize, max_batch_size: usize, ) -> Self

2026-09-25: Every buffer size, in bytes. max_seq_len and kv_block_size set the block-table width in scratch (max_seq_len / kv_block_size + 1 blocks, 256 when kv_block_size is 0) and the QSA score width; max_batch_size sets the decode-metadata rows.

Trait Implementations

impl Clone for BufferSizes

fn clone(&self) -> BufferSizes

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for BufferSizes

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.