Module cublaslt 

2026-09-25: cuBLASLt FFI for out[M,N] = act[M,K] @ weight[N,K]ᵀ: BF16 GEMMs with BF16 or FP32 output here, FP8 E4M3 GEMMs in fp8, and the FP8 scale-tensor layout math in scale_layout.

Owner: gpu-runtime (cuBLASLt wrapper). Invariants:

• CTX holds a single cuBLASLt handle and 64 MiB workspace for the process, allocated with cuMemAlloc_v2, outside the backend’s allocation ledger.
• Every GEMM here passes the whole workspace to cublasLtMatmul, so two GEMMs must not run concurrently.

Modules

scale_layout

2026-09-25: Index math for the FP32 scale tensors of the block-scaled FP8 GEMM (super::fp8_gemm_act_weight_t_blkscaled): the activation scales in the layout it passes to cuBLASLt, in the quantizer’s layout, and the weight-scale stride check.

Functions

bf16_gemm_act_weight_t

2026-09-25: Row-major out[M,N] = act[M,K] @ weight[N,K]ᵀ, all BF16, FP32 accumulation. In cuBLASLt’s column-major terms: D[N,M] = opT(weightᶜ[K,N]) · opN(actᶜ[K,M]).

bf16_gemm_act_weight_t_f32_out

2026-09-25: bf16_gemm_act_weight_t with an FP32 out. Only the D layout’s type differs; both use CUBLAS_COMPUTE_32F.

fp8_gemm_act_weight_t_blkscaled

2026-09-25: FP8 E4M3 out[M,N] = act[M,K] @ weight[N,K]ᵀ, BF16 output, with FP32 block scales: BLK128x128_32F on the weight (cuBLASLt’s A) and VEC128_32F on the activation (B). The scale layouts are in super::scale_layout:

fp8_gemm_act_weight_t_blkscaled_ldc

2026-09-25: fp8_gemm_act_weight_t_blkscaled with output rows ldc BF16 elements apart: D is column-major [N, m] with leading dimension ldc, i.e. row-major [m, N] with row pitch ldc. The multi-seq decode Q/K/V projections use it to write each row into its sequence’s slot of the QKV buffer (w8a8_decode.rs qkv_decode_w8a8_plans, ldc = per_seq_qkv in BF16 elements).

fp8_gemm_act_weight_t_rowwise

2026-09-25: FP8 E4M3 out[M,N] = act[M,K] @ weight[N,K]ᵀ, BF16 output, with FP32 OUTER_VEC scales on both operands: weight_scale[N] per weight row (cuBLASLt’s A) and act_scale[M] per token (B).

prewarm

2026-09-25: Run one 64x64x64 BF16 GEMM on stream at model load (serve_load.rs), so the first cuBLASLt call, including ctx’s cublasLtCreate and workspace allocation, happens before the first request.