Module quantized 

2026-09-25: Device-resident weight types (QuantizedWeight, DenseWeight, Fp8Weight, PackedQ2Weight) and the WeightQuantFormat tag.

Owner: model-layers (weight loading). Invariants: none beyond the types.

Structs

DenseWeight

2026-09-25: Device pointer to an unquantized weight: BF16 unless the loader that built it says otherwise (for example dense_keep_f32).

Fp8DenseWeight

2026-09-25: FP8 E4M3 weight quantized from BF16 at load, with one f32 scale per row.

Fp8Weight

2026-09-25: FP8 E4M3 weight with its scale. The layout of row_scale depends on scale_format:

Fp8WeightTransposed

2026-09-25: The transpose of a block-scaled Fp8Weight, built by Fp8Weight::transpose_for_gemm for the prefill GEMM.

PackedQ2Weight

2026-09-25: Keep-packed ternary Q2_0 weight: one buffer of raw block_q2_0 blocks ([fp16 d][group/4 bytes of 2-bit codes], value = (code - 1) * d), row-major over [n, k], with the scale inline and no separate scale tensor. Built from a WeightDtype::PackedQ2_0 store tensor, which the GGUF loader produces under METRALE_GGUF_NATIVE_Q2=1. The store owns the buffer; this struct only holds the pointer.

QuantizedWeight

2026-09-25: 4-bit quantized weight: packed E2M1 data, per-group scales and a host-side per-tensor scale.

Enums

WeightQuantFormat

2026-09-25: The format of a weight buffer in device memory, as opposed to the on-disk format (Nvfp4Variant). A kernel call site checks it with WeightQuantFormat::expect, because a kernel cannot tell one layout from another by its pointer.