pub struct Fp8KvCalibration { /* private fields */ }Expand description
2026-09-25: The calibrator of one attention layer. observe takes &self;
the state sits behind a Mutex.
Implementations§
Source§impl Fp8KvCalibration
impl Fp8KvCalibration
Sourcepub fn new(
attn_layer_idx: usize,
window_tokens: usize,
headroom: f32,
gpu: &dyn GpuBackend,
) -> Result<Self>
pub fn new( attn_layer_idx: usize, window_tokens: usize, headroom: f32, gpu: &dyn GpuBackend, ) -> Result<Self>
2026-09-25: Build the calibrator for attention layer attn_layer_idx.
window_tokens is --fp8-kv-calibration-tokens, clamped to
MAX_STAGED_TOKENS with a warning. 1 freezes on the first observe and
stages nothing. The attention initializer builds no calibrator for 0
(qwen3_attention/init.rs).
headroom is --fp8-kv-headroom; a value below 1.0 is clamped to 1.0
with a warning, because it would clip the window’s own amax.
Errors if the bf16_absmax kernel is missing or the 8-byte absmax
buffer cannot be allocated or zeroed.
Sourcepub fn is_calibrating(&self) -> bool
pub fn is_calibrating(&self) -> bool
2026-09-25: True until the scales freeze.
Sourcepub fn scales(&self) -> (f32, f32)
pub fn scales(&self) -> (f32, f32)
2026-09-25: (k_scale, v_scale): PROVISIONAL_SCALE inside the window,
the frozen scales after it.
Sourcepub fn observe(
&self,
gpu: &dyn GpuBackend,
k_data: DevicePtr,
v_data: DevicePtr,
num_tokens: u32,
num_kv_heads: u32,
head_dim: u32,
stream: u64,
target: &Fp8KvWriteTarget,
) -> Result<()>
pub fn observe( &self, gpu: &dyn GpuBackend, k_data: DevicePtr, v_data: DevicePtr, num_tokens: u32, num_kv_heads: u32, head_dim: u32, stream: u64, target: &Fp8KvWriteTarget, ) -> Result<()>
2026-09-25: Fold this batch’s BF16 K/V ([num_tokens, num_kv_heads, head_dim])
into the window.
Inside the window it reduces the absmax on the device, synchronizes the
stream to read it back, and stages the batch. The observe that reaches
the window freezes the scales and replays the staged batches into
target’s pools at them. Call it after the K/V projections and before
the cache write, which then uses Self::scales. After the freeze it
reduces only when CalibrationState::should_observe asks, for the
opt-in EMA path.