int8_gemm_faith2_prefill

Function int8_gemm_faith2_prefill 

Source
pub fn int8_gemm_faith2_prefill(
    gpu: &dyn GpuBackend,
    faith2_kernel: KernelHandle,
    requant_a_kernel: KernelHandle,
    a_bf16: DevicePtr,
    w_i8: DevicePtr,
    w_scale: DevicePtr,
    a_i8_scratch: DevicePtr,
    a_scale_scratch: DevicePtr,
    out: DevicePtr,
    m: u32,
    n: u32,
    k: u32,
    stream: u64,
) -> Result<()>
Expand description

2026-09-25: int8 prefill GEMM: requantize the BF16 activations to int8 with one F32 scale per 32 elements (requant_a_kernel), then multiply them with the int8 weight from requant_w_nvfp4_int8 (faith2_kernel). Both launches go on stream.

a_bf16 [M, K] BF16, w_i8 [N, K] int8, w_scale [N, K/32] F32, out [M, N] BF16. a_i8_scratch and a_scale_scratch are caller-owned, of at least M*K and M*(K/32)*4 bytes.