Trait QuantWeights 

pub trait QuantWeights: Send + Sync {
    // Required methods
    fn out_features(&self) -> u32;
    fn in_features(&self) -> u32;
    fn gemv(
        &self,
        gpu: &dyn GpuBackend,
        x: DevicePtr,
        y: DevicePtr,
        stream: u64,
    ) -> Result<()>;

    // Provided methods
    fn gemv_gate_up_with(
        &self,
        other: &Self,
        gpu: &dyn GpuBackend,
        x: DevicePtr,
        gate_y: DevicePtr,
        up_y: DevicePtr,
        stream: u64,
    ) -> Result<()>
       where Self: Sized { ... }
    fn gemv_silu_gate(
        &self,
        _gpu: &dyn GpuBackend,
        _gate: DevicePtr,
        _up: DevicePtr,
        _y: DevicePtr,
        _stream: u64,
    ) -> Result<()> { ... }
    fn gemv_silu_gate_resid(
        &self,
        _gpu: &dyn GpuBackend,
        _gate: DevicePtr,
        _up: DevicePtr,
        _x_resid: DevicePtr,
        _y: DevicePtr,
        _stream: u64,
    ) -> Result<()> { ... }
}

2026-09-25: A quantised [N, K] weight that runs matvecs on a GpuBackend. The one implementation, for MlxInt8Weight, is in this file.

Required Methods

fn out_features(&self) -> u32

2026-09-25: Output dimension N of the [N, K] weight.

fn in_features(&self) -> u32

2026-09-25: Input dimension K of the [N, K] weight.

fn gemv( &self, gpu: &dyn GpuBackend, x: DevicePtr, y: DevicePtr, stream: u64, ) -> Result<()>

2026-09-25: Decode matvec y = self @ x.

x is a BF16 buffer of length in_features(); y must hold at least out_features() BF16 slots.

Provided Methods

fn gemv_gate_up_with( &self, other: &Self, gpu: &dyn GpuBackend, x: DevicePtr, gate_y: DevicePtr, up_y: DevicePtr, stream: u64, ) -> Result<()>

where Self: Sized,

2026-09-25: Dual-output GEMV over one input: gate_y = self @ x and up_y = other @ x. The default is two gemv calls. The MlxInt8Weight impl overrides it with the single mlx_int8_gemv_gate_up launch.

where Self: Sized keeps this method off the dyn QuantWeights surface; the forward modules call it through a generic parameter.

fn gemv_silu_gate( &self, _gpu: &dyn GpuBackend, _gate: DevicePtr, _up: DevicePtr, _y: DevicePtr, _stream: u64, ) -> Result<()>

2026-09-25: Fused FFN tail y = self @ (silu(gate) ⊙ up). The default returns an error; an implementation overrides it with a fused kernel or its own composition.

fn gemv_silu_gate_resid( &self, _gpu: &dyn GpuBackend, _gate: DevicePtr, _up: DevicePtr, _x_resid: DevicePtr, _y: DevicePtr, _stream: u64, ) -> Result<()>

2026-09-25: Self::gemv_silu_gate with the residual add folded in: y[n] = x_resid[n] + sum_k self[n, k] * (silu(gate[k]) ⊙ up[k]).

The default returns an error.

Implementations on Foreign Types

impl QuantWeights for MlxInt8Weight

fn out_features(&self) -> u32

fn in_features(&self) -> u32

fn gemv( &self, gpu: &dyn GpuBackend, x: DevicePtr, y: DevicePtr, stream: u64, ) -> Result<()>

fn gemv_gate_up_with( &self, other: &Self, gpu: &dyn GpuBackend, x: DevicePtr, gate_y: DevicePtr, up_y: DevicePtr, stream: u64, ) -> Result<()>

fn gemv_silu_gate( &self, gpu: &dyn GpuBackend, gate: DevicePtr, up: DevicePtr, y: DevicePtr, stream: u64, ) -> Result<()>

fn gemv_silu_gate_resid( &self, gpu: &dyn GpuBackend, gate: DevicePtr, up: DevicePtr, x_resid: DevicePtr, y: DevicePtr, stream: u64, ) -> Result<()>

Implementors