pub struct MtpWeights {Show 18 fields
pub pre_fc_norm_embedding: DenseWeight,
pub pre_fc_norm_hidden: DenseWeight,
pub fc: DenseWeight,
pub input_layernorm: DenseWeight,
pub q_proj: DenseWeight,
pub k_proj: DenseWeight,
pub v_proj: DenseWeight,
pub o_proj: DenseWeight,
pub q_norm: DenseWeight,
pub k_norm: DenseWeight,
pub post_attn_layernorm: DenseWeight,
pub moe_gate: DenseWeight,
pub shared_expert: DenseExpertWeight,
pub shared_expert_gate: DenseWeight,
pub experts: Vec<DenseExpertWeight>,
pub dense_ffn: Option<DenseExpertWeight>,
pub norm: DenseWeight,
pub fp8_experts: Option<MtpFp8Experts>,
}Expand description
2026-09-25: MTP (multi-token prediction) head weights: one decoder layer
plus the concat projection fc.
Every DenseWeight here is BF16 (load_mtp dequantizes FP8 or NVFP4
projections). MtpHead::new quantizes the projections when it builds the head.
Fields§
§pre_fc_norm_embedding: DenseWeight2026-09-25: RMSNorm weight for the token embedding before the concat.
2026-09-25: RMSNorm weight for the target hidden state before the concat.
fc: DenseWeight2026-09-25: Concat projection [hidden_size, 2*hidden_size] BF16.
input_layernorm: DenseWeight2026-09-25: RMSNorm weight before attention.
q_proj: DenseWeight2026-09-25: Attention projections, BF16.
k_proj: DenseWeight§v_proj: DenseWeight§o_proj: DenseWeight§q_norm: DenseWeight§k_norm: DenseWeight§post_attn_layernorm: DenseWeight2026-09-25: RMSNorm weight after attention (fused with the residual add).
moe_gate: DenseWeight2026-09-25: MoE router gate, BF16. NULL when dense_ffn is Some.
2026-09-25: Shared expert, BF16. NULL fields when dense_ffn is Some.
2026-09-25: Shared-expert gate [1, hidden_size] BF16. NULL when dense_ffn is Some.
experts: Vec<DenseExpertWeight>2026-09-25: One BF16 entry per expert. Empty when dense_ffn is Some.
dense_ffn: Option<DenseExpertWeight>2026-09-25: Dense FFN (gate_proj, up_proj, down_proj) of an MTP
head without a router. When Some, MtpHead::new builds no MoE and
the MoE fields above are unused.
norm: DenseWeight2026-09-25: Final RMSNorm weight.
fp8_experts: Option<MtpFp8Experts>2026-09-25: The routed and shared experts as the checkpoint’s own FP8
block-scaled tables (mtp.layers.0.mlp.experts.{e}.*_proj.{weight,weight_scale_inv}).
When Some, experts and shared_expert above are fresh BF16 dequants
of the same tensors; a head that builds its MoE from these tables calls
Self::release_bf16_expert_dequants. None when the experts are not
FP8 on disk, are stacked, or the variant is Bf16Raw.
Implementations§
Source§impl MtpWeights
impl MtpWeights
Sourcepub fn release_bf16_expert_dequants(
&mut self,
gpu: &dyn GpuBackend,
) -> Result<()>
pub fn release_bf16_expert_dequants( &mut self, gpu: &dyn GpuBackend, ) -> Result<()>
2026-09-25: Free the BF16 dequants of the routed and shared experts once
an FP8 MoeLayer serves them. Returns an error when
Self::fp8_experts is None. load_mtp attaches the tables only for
unstacked FP8 experts and a variant other than Bf16Raw; every expert
entry is then a fresh dequant allocation, never a store pointer. Leaves
experts empty and shared_expert NULL.