pub fn int8_gemm_faith2_prefill(
gpu: &dyn GpuBackend,
faith2_kernel: KernelHandle,
requant_a_kernel: KernelHandle,
a_bf16: DevicePtr,
w_i8: DevicePtr,
w_scale: DevicePtr,
a_i8_scratch: DevicePtr,
a_scale_scratch: DevicePtr,
out: DevicePtr,
m: u32,
n: u32,
k: u32,
stream: u64,
) -> Result<()>Expand description
2026-09-25: int8 prefill GEMM: requantize the BF16 activations to int8
with one F32 scale per 32 elements (requant_a_kernel), then multiply them
with the int8 weight from requant_w_nvfp4_int8 (faith2_kernel). Both
launches go on stream.
a_bf16 [M, K] BF16, w_i8 [N, K] int8, w_scale [N, K/32] F32,
out [M, N] BF16. a_i8_scratch and a_scale_scratch are caller-owned,
of at least M*K and M*(K/32)*4 bytes.