decode_w8a8_quant_act

Function decode_w8a8_quant_act 

Source
pub fn decode_w8a8_quant_act(
    gpu: &dyn GpuBackend,
    scratch: &DecodeW8a8Scratch,
    act_bf16: DevicePtr,
    rows: u32,
    k: u32,
    stream: u64,
) -> Result<()>
Expand description

2026-09-25: Quantize BF16 act[rows, k] into the scratch: FP8 E4M3 bytes and one FP32 scale per token and 128 of K, with the pad rows rows..ceil16(rows) zeroed, and the scales transposed for cuBLASLt when the layout is K-major.

Separate from the GEMM so projections over one activation share one quantization; the attention Q/K/V do.

The pad rows’ FP8 bytes are zeroed as well as their scales: a zero scale alone is not enough, because NaN * 0.0 is NaN.