Module fp8_residency 

2026-09-25: The native-FP8 dense loader’s derived-copy decision table and the shape arithmetic that prices it. 2026-09-25: Which derived weight copies the native-FP8 dense route builds, and a tally of the derived bytes the loader kept, skipped and freed.

Owner: model-arch weight loader (Qwen3.5 dense). Invariants:

• DenseFp8Plan::resolve reads no environment and allocates nothing.
• With keep_nvfp4 set, the plan builds every NVFP4 copy, and every FP8 twin on a layer with an FP8 attention overlay.

With the native FP8 overlay installed, the dense FFN reads no NVFP4 weight: DenseFfnLayer::forward and forward_prefill_inner return inside their fp8_weights arm, forward_k2, forward_k3 and forward_km send an FP8 layer to forward_prefill (native_small_batch_uses_prefill), and the prefill w8_gemm! is given None for every transposed operand, so each of its arms reads the untransposed E4M3 weight. METRALE_FFN_W8A16_ONLY only turns off the W8A8 arm of that macro, so it is not an input here.

The loader runs before a ForwardContext exists, so the route comes from GemmDispatch::from_env, the constructor the model uses for the context’s dispatch. A lever that can send a native-FP8 layer to an NVFP4 kernel must be an input of DenseFp8Plan::resolve as well as of that dispatch site, or the loader skips a copy the site reads. METRALE_DENSE_FP8_KEEP_NVFP4 builds every copy while such a gap is diagnosed.

Structs

DenseFp8Inputs

2026-09-25: The inputs of DenseFp8Plan::resolve. A plain struct, so the decision-table tests set every clause without the process environment, which keep_nvfp4_fallback caches in a OnceLock.

DenseFp8Plan

2026-09-25: Which derived copies one dense layer builds beyond the checkpoint bytes. Each field means “build this copy”.

DerivedResidency

2026-09-25: Running tally, kept by the loader as it goes, of the derived (non-checkpoint) device bytes it built, freed and declined to build.

RouteEnv

2026-09-25: The environment half of DenseFp8Inputs. load_layers resolves it once, so every layer of one load is planned from the same values.

TwinsBuilt

2026-09-25: Which derived families the loader built, named in the summary line.

Functions

attn_fp8_twin_bytes

2026-09-25: Bytes of the FP8 prefill twins want selects, for one full-attention layer.

attn_nvfp4_bytes

2026-09-25: NVFP4 bytes of one full-attention layer: q/k/v/o, each with a transposed twin, plus the fused [q|k|v] transposed twin (transpose_concat_for_gemm).

dense_ffn_nvfp4_bytes

2026-09-25: NVFP4 bytes of one dense-FFN layer: gate, up and down, each with a transposed twin of the same size.

ffn_gateup_fused_bytes

2026-09-25: Bytes of one fused dense-FFN gate+up weight: the two [inter, hidden] E4M3 blocks side by side, plus their two FP32 block-scale grids side by side. Two grids, not one grid over 2 * inter, because the concat copies the two source grids.

ffn_gateup_fused_parts

2026-09-25: ffn_gateup_fused_bytes as (weight bytes, scale-grid bytes). The loader adopts the two buffers separately.

fp8_twin_bytes

2026-09-25: Bytes of one FP8 transposed twin: n * k E4M3 bytes plus the FP32 block-scale grid over 128x128 blocks, the two allocations of Fp8Weight::transpose_for_gemm.

keep_nvfp4_fallback

2026-09-25: Whether METRALE_DENSE_FP8_KEEP_NVFP4 is set, which makes the plan build every NVFP4 copy even where the dispatch cannot reach it. Presence, like METRALE_FFN_W8A16_ONLY: any value, including empty and 0, turns it on. Read once per process.

nvfp4_bytes

2026-09-25: Bytes of one NVFP4 QuantizedWeight: n * k / 2 packed E2M1 bytes plus n * k / 16 group-scale bytes, the two allocations of quantize_to_nvfp4 and of QuantizedWeight::transpose_for_gemm_gs with 16-wide groups.