Struct MlxInt8Weight 

pub struct MlxInt8Weight {
    pub packed: DevicePtr,
    pub scales: DevicePtr,
    pub biases: DevicePtr,
    pub out_features: u32,
    pub in_features: u32,
    pub group_size: u32,
}

2026-09-25: One 8-bit affine linear weight resident on the GPU. There is no Drop: the owner frees the three buffers with Self::release.

Fields

packed: DevicePtr

2026-09-25: [out_features, in_features / 4] packed bytes (uint32 words).

scales: DevicePtr

2026-09-25: [out_features, in_features / group_size] per-group BF16 scales.

biases: DevicePtr

2026-09-25: [out_features, in_features / group_size] per-group BF16 biases.

out_features: u32

in_features: u32

group_size: u32

Implementations

impl MlxInt8Weight

pub fn load( gpu: &dyn GpuBackend, st: &SafeTensors<'_>, base: &str, group_size: u32, ) -> Result<Self>

2026-09-25: Load the {base}.weight, {base}.scales and {base}.biases tensors from a parsed safetensors blob and upload them. Fails on a missing tensor, a weight that is not 2-D U32, scales or biases that are not BF16 of shape [out_features, in_features / group_size], or an in_features that group_size does not divide.

pub fn dequantize_to( &self, gpu: &dyn GpuBackend, out: DevicePtr, stream: u64, ) -> Result<()>

2026-09-25: Write the full dequantised weight as BF16 to out, which must hold at least out_features * in_features * 2 bytes. Runs mlx_int8_dequant.

pub fn gemv( &self, gpu: &dyn GpuBackend, x: DevicePtr, y: DevicePtr, stream: u64, ) -> Result<()>

2026-09-25: y = W x, with x BF16 [in_features] and y a BF16 buffer of at least out_features elements. Runs mlx_int8_gemv.

pub fn gemv_silu_gate_resid( &self, gpu: &dyn GpuBackend, gate: DevicePtr, up: DevicePtr, x_resid: DevicePtr, y: DevicePtr, stream: u64, ) -> Result<()>

2026-09-25: Self::gemv_silu_gate plus the residual, in one launch: y[n] = x_resid[n] + sum_k W[n, k] * (silu(gate[k]) * up[k]).

pub fn gemv_silu_gate( &self, gpu: &dyn GpuBackend, gate: DevicePtr, up: DevicePtr, y: DevicePtr, stream: u64, ) -> Result<()>

2026-09-25: y = W (silu(gate) * up) in one launch of mlx_int8_gemv_silu_gate; silu(gate) * up is never written to memory.

pub fn gemm( &self, gpu: &dyn GpuBackend, x: DevicePtr, y: DevicePtr, m: u32, stream: u64, ) -> Result<()>

2026-09-25: Y = X W^T, with X BF16 [m, in_features] and Y BF16 [m, out_features]. Runs mlx_int8_gemm, one thread per output element.

pub fn release(&self, gpu: &dyn GpuBackend) -> Result<()>

2026-09-25: Free the three GPU buffers. DevicePtr is a plain handle, so nothing frees them automatically; call this once.