Struct LayerKvCache 

pub struct LayerKvCache {
    pub k: DevicePtr,
    pub v: DevicePtr,
    pub capacity: u32,
    pub dtype: MetalKvDtype,
    pub k_scales: Option<DevicePtr>,
    pub v_scales: Option<DevicePtr>,
}

2026-09-25: KV cache of one full-attention layer, for one sequence.

dtype selects the storage format. A quantized side holds packed data in k/v and its per-16-element group scales in k_scales/v_scales (BF16 for Turbo8, FP8 E4M3 otherwise).

Fields

k: DevicePtr

v: DevicePtr

capacity: u32

2026-09-25: Capacity in tokens: the max_seq given to Self::alloc.

dtype: MetalKvDtype

k_scales: Option<DevicePtr>

2026-09-25: Group-scale buffers, Some only for a quantized side: both for Turbo8/4/3/2, V only for the Bf16KTurbo*V formats, neither for Bf16.

v_scales: Option<DevicePtr>

Implementations

impl LayerKvCache

pub fn alloc( gpu: &dyn GpuBackend, dtype: MetalKvDtype, max_seq: u32, kv_dim: u32, ) -> Result<Self>

2026-09-25: Allocates a cache of max_seq tokens by kv_dim elements in the given format. Panics for a turbo format when kv_dim is not a multiple of 16.