apply_lora_bgmv

Function apply_lora_bgmv 

Source
pub fn apply_lora_bgmv(
    gpu: &dyn GpuBackend,
    kernels: &LoraKernels,
    route: &LoraRoute,
    x: DevicePtr,
    base_out: DevicePtr,
    seq_slot: DevicePtr,
    n: u32,
    x_row_stride: u32,
    out_row_stride: u32,
    lora_xa: DevicePtr,
    stream: u64,
) -> Result<()>
Expand description

2026-09-25: Per-request routed LoRA delta over n rows, each naming its adapter slot in seq_slot[n] (i32; a negative slot gets no delta): out[i, :] += scale_s * (x[i, :] @ A_s^T) @ B_s^T with s = seq_slot[i].

Two launches (kernels/gb10/common/lora_bgmv.cu): the shrink writes BF16 xa, and the expand rounds each delta to BF16 before adding scale_table[s] * delta in FP32, the order of bf16_scaled_add. A slot whose table entry is 0 does not adapt this module. Both stages run at route.max_rank.

Strides are in elements: x_row_stride between x rows (at least k_in) and out_row_stride between base_out rows (at least n_out), so the fold can land inside an interleaved buffer. lora_xa must hold n * max_rank BF16. The launch arguments follow the kernels’ parameter order; the launch does not check their types.