Struct KvCacheConfig 

pub struct KvCacheConfig {
    pub block_size: usize,
    pub num_kv_heads: usize,
    pub head_dim: usize,
    pub num_layers: usize,
    pub dtype: KvCacheDtype,
    pub layer_dtypes: Vec<KvCacheDtype>,
    pub layer_dims: Vec<(usize, usize)>,
    pub cache_blocks_per_seq: Option<u32>,
}

2026-09-25: Geometry and formats of the paged KV cache.

Fields

block_size: usize

2026-09-25: Tokens per block.

num_kv_heads: usize

2026-09-25: KV heads per layer, where layer_dims does not override it.

head_dim: usize

2026-09-25: Head dimension, where layer_dims does not override it.

num_layers: usize

2026-09-25: Attention layers that get a K pool and a V pool.

dtype: KvCacheDtype

2026-09-25: Format of every layer that layer_dtypes does not cover.

layer_dtypes: Vec<KvCacheDtype>

2026-09-25: Per-layer format: layer i uses layer_dtypes[i] when i < layer_dtypes.len(), else dtype.

layer_dims: Vec<(usize, usize)>

2026-09-25: Per-layer (num_kv_heads, head_dim), by the same rule as layer_dtypes, for models whose attention layers differ in shape (the Gemma-4 loader’s kv_layer_dims). Pool sizes and block strides use it.

cache_blocks_per_seq: Option<u32>

2026-09-25: Per-sequence cap on HBM-resident blocks under --high-speed-swap. With Some(N) the model engine slides a sequence’s window at N blocks (ensure_blocks_through_decode); None means no cap.

Implementations

impl KvCacheConfig

pub fn dtype_for_layer(&self, layer_idx: usize) -> KvCacheDtype

2026-09-25: The format of attention layer layer_idx.

pub fn v_block_bytes_dims( &self, dtype: KvCacheDtype, nkv: usize, hd: usize, ) -> usize

2026-09-25: V-side bytes per block: block_bytes_dims of dtype.kv_pair().1.

pub fn k_block_bytes_dims( &self, dtype: KvCacheDtype, nkv: usize, hd: usize, ) -> usize

2026-09-25: K-side bytes per block: block_bytes_dims of dtype.kv_pair().0 (for Bf16KTurbo3V, the BF16 size).

pub fn k_block_bytes_for_layer(&self, layer_idx: usize) -> usize

2026-09-25: K-side bytes per block of attention layer layer_idx, from its dims and the K side of its format. PagedKvCache::new sizes the K pool with it.

pub fn v_block_bytes_for_layer(&self, layer_idx: usize) -> usize

2026-09-25: V-side bytes per block of attention layer layer_idx. Equals k_block_bytes_for_layer for a symmetric format.

pub fn block_bytes(&self) -> usize

2026-09-25: Bytes of one side of one block with the uniform dtype and the global dims. For a *K*V dtype this is the K side.

pub fn dims_for_layer(&self, layer_idx: usize) -> (usize, usize)

2026-09-25: (num_kv_heads, head_dim) of attention layer layer_idx: layer_dims[layer_idx] when present, else the global values.

pub fn block_bytes_for_layer(&self, layer_idx: usize) -> usize

2026-09-25: Bytes of one side of one block of attention layer layer_idx, from its own dims and format. For a *K*V format this is the K side.

pub fn block_bytes_kv(&self) -> usize

2026-09-25: Twice block_bytes(). That is K + V for a symmetric uniform dtype, but twice the K side for a *K*V dtype.

pub fn block_bytes_kv_all_layers(&self) -> usize

2026-09-25: K plus V bytes of one block index summed over all layers, per layer’s own dims and K and V formats.

pub fn cache_stride_elements(&self) -> usize

2026-09-25: Elements per block and side, from the global dims: block_size * num_kv_heads * head_dim.

pub fn nvfp4_data_bytes(&self) -> usize

2026-09-25: NVFP4 data bytes per block (two E2M1 values per byte), from the global dims.

pub fn nvfp4_scale_bytes(&self) -> usize

2026-09-25: NVFP4 scale bytes per block (one FP8 byte per group), from the global dims.

pub fn turbo4_data_bytes(&self) -> usize

2026-09-25: Turbo4 data bytes per block, equal to the NVFP4 value.

pub fn turbo4_scale_bytes(&self) -> usize

2026-09-25: Turbo4 scale bytes per block, equal to the NVFP4 value.

pub fn turbo3_data_bytes(&self) -> usize

2026-09-25: Turbo3 data bytes per block (8 values in 3 bytes).

pub fn turbo3_scale_bytes(&self) -> usize

2026-09-25: Turbo3 scale bytes per block, equal to the NVFP4 value.

pub fn turbo2_data_bytes(&self) -> usize

2026-09-25: Turbo2 data bytes per block (4 values per byte).

pub fn turbo2_scale_bytes(&self) -> usize

2026-09-25: Turbo2 scale bytes per block, equal to the NVFP4 value.

pub fn turbo8_data_bytes(&self) -> usize

2026-09-25: Turbo8 data bytes per block (one FP8 byte per element).

pub fn turbo8_scale_bytes(&self) -> usize

2026-09-25: Turbo8 scale bytes per block: one 2-byte BF16 scale per group, twice the NVFP4 value.