Enum KvCacheDtype 

pub enum KvCacheDtype {
    Bf16,
    Fp8,
    Nvfp4,
    Turbo4,
    Turbo3,
    Turbo2,
    Turbo8,
    Turbo4KTurbo3V,
    Turbo4KTurbo8V,
    Turbo3KTurbo8V,
    Bf16KTurbo4V,
    Bf16KTurbo3V,
    Fp8KTurbo4V,
    Fp8KTurbo3V,
    Bf16KTurbo2V,
    Fp8KTurbo2V,
}

2026-09-25: KV cache storage format. A symmetric variant uses one format for K and V; a *K*V variant names the K format, then the V format (kv_pair).

Variants

Bf16

2026-09-25: BF16, 2 bytes per element.

Fp8

2026-09-25: FP8 E4M3, 1 byte per element, with one scale per tensor (the k_scale/v_scale arguments of reshape_and_cache_flash_fp8).

Nvfp4

2026-09-25: E2M1 values, two per byte, plus one FP8 E4M3 scale byte per 16-element group: 4.5 bits per element.

Turbo4

2026-09-25: WHT-rotated 16-level Lloyd-Max codebook (TURBO4_LUT) in the Nvfp4 byte layout.

Turbo3

2026-09-25: WHT-rotated 8-level Lloyd-Max codebook (TURBO3_LUT), 8 values in 3 bytes plus one FP8 scale byte per group: 3.5 bits per element, 22% smaller than Turbo4.

Turbo2

2026-09-25: WHT-rotated 4-level codebook, 4 values per byte plus one FP8 scale byte per group: 2.5 bits per element, 6.4x smaller than Bf16. The server’s auto_high_precision_layers gives this dtype a larger high-precision boundary than the other symmetric turbo dtypes.

Turbo8

2026-09-25: WHT-rotated FP8 E4M3 data plus one BF16 scale per group: 9 bits per element, 12.5% more than Fp8.

Turbo4KTurbo3V

2026-09-25: K Turbo4, V Turbo3: 8 bits per K+V element pair, against 9 for Turbo4.

Turbo4KTurbo8V

2026-09-25: K Turbo4, V Turbo8: 13.5 bits per K+V element pair, against 18 for Turbo8.

Turbo3KTurbo8V

2026-09-25: K Turbo3, V Turbo8.

Bf16KTurbo4V

2026-09-25: K Bf16, V Turbo4.

Bf16KTurbo3V

2026-09-25: K Bf16, V Turbo3.

Fp8KTurbo4V

2026-09-25: K Fp8, V Turbo4.

Fp8KTurbo3V

2026-09-25: K Fp8, V Turbo3.

Bf16KTurbo2V

2026-09-25: K Bf16, V Turbo2.

Fp8KTurbo2V

2026-09-25: K Fp8, V Turbo2.

Implementations

impl KvCacheDtype

pub const ALL: [KvCacheDtype; 16]

2026-09-25: Every variant, in declaration order. Nothing checks that a new variant is added here; add it together with its KvCacheDtype::name arm, which the compiler does require.

pub const fn name(self) -> &'static str

2026-09-25: The canonical --kv-cache-dtype spelling. Display delegates here, so a listed value is the string FromStr accepts; the tests below check that round trip.

impl KvCacheDtype

pub fn kv_pair(self) -> (KvCacheDtype, KvCacheDtype)

2026-09-25: The (K, V) formats. A symmetric variant returns (self, self).

pub fn is_wht_rotated(self) -> bool

2026-09-25: True for the symmetric turbo formats, whose cache contents are in the WHT-rotated basis: write_kv_cache.rs runs wht_bf16_inplace on K and V before the turbo write. The decode attention calls it on each side of kv_pair(), applying the WHT to Q when K is rotated and the inverse WHT to the output when V is rotated. A *K*V variant itself returns false, so call it on a side.

pub fn is_asymmetric(self) -> bool

2026-09-25: True if K and V use different formats.

Trait Implementations

impl Clone for KvCacheDtype

fn clone(&self) -> KvCacheDtype

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for KvCacheDtype

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for KvCacheDtype

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl FromStr for KvCacheDtype

type Err = Error

The associated error which can be returned from parsing.

fn from_str(s: &str) -> Result<Self>

Parses a string s to return a value of this type.

impl PartialEq for KvCacheDtype

fn eq(&self, other: &KvCacheDtype) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Copy for KvCacheDtype

impl Eq for KvCacheDtype

impl StructuralPartialEq for KvCacheDtype