Module weights

Module weights 

Source
Expand description

2026-10-02: Weight bytes per node: what the checkpoint stores for it (its declared formats, scales included) and every load-time copy the loader derives from it (super::copies).

Owner: metrale-circuit (memory). Invariants:

  • A weight is out x k: k the node’s first input edge’s dim, out the sum of its output edges’ dims. Routed experts hold experts such weights, whatever the rows routed to them.
  • The embedding table is vocab x hidden at its output edge’s format, counted once: the draft head reads the main table.
  • Stored bytes are the declared circuit’s (the checkpoint’s own formats); served bytes appear only through copy rules, since the loader keeps every stored tensor resident.
  • A tensor is stored once: a node whose binding another node already holds stores nothing, though copies may still derive from it. The draft head’s lm_head reads the target’s table (no checkpoint ships its own; Qwen’s circuit binds the runtime copy as mtp.lm_head), so it stores nothing either.

Structs§

DerivedCopy
2026-10-02: One derived copy of a node’s weight.
NodeWeights
2026-10-02: One node’s weights.

Functions§

node_weights
2026-10-02: Every weight of served (its embedding and every weight-reading node), stored at the formats declared gives the node of the same id, with the copies rules derive under settings.