Module lora 

2026-09-25: PEFT LoRA adapters: audit, pack into the rank-padded slot pool and the expert pool, slot routing and swap, and token overlays. “LoRA” here means adapters; kv_lora_rank / q_lora_rank in the model config are MLA dimensions, unrelated.

Owner: model-layers (lora). Invariants: none beyond the types.

Modules

rdma_stage

2026-09-25: The model-layers side of the RDMA adapter swap: fetch a peer-staged adapter’s manifest, map each tensor to its landing offset in a pool slot, and rebuild the slot’s pairs after model-engine’s swap_lora_slot_from_peer has landed them with RdmaLoraLoader.

Structs

AdapterSlot

2026-09-25: One pool slot: the resident adapter’s name and config, and its pairs, which point into this slot’s region of the shared pool. layers is indexed by global layer and has num_hidden_layers entries. An empty name marks a placeholder that holds no adapter.

EmbedOverlay

2026-09-25: One adapter slot’s embed overlay: rows ([n_override, h] BF16) in the order of ids_dev (ascending u32 vocab ids), and slot_map (i32 [vocab], the row index for an overridden id, else -1), which the embed kernel indexes by token id. lmhead is Some only for an untied head with its own overlay tensors.

ExpertLoraLayer

2026-09-25: One MoE layer’s routed-expert pairs, keyed by (expert_index, projection); only adapted experts have entries.

ExpertWork

2026-09-25: One expert’s contiguous block of sorted rows. The planner, expert_delta_workitems, emits none with rows == 0.

LmHeadOverlay

2026-09-25: The overlay of an untied lm_head: rows ([n_override, h] BF16) and their vocab ids. The lm_head kernel writes dot(hidden, row) to each overridden id’s logit.

LoraAdapterInput

2026-09-25: One adapter to pack. store holds its tensors on the device as BF16: metrale_model_weights::weights::adapter::load_adapter_safetensors converts F16 and F32 on the host.

LoraLayerWeights

2026-09-25: One layer’s adapted modules, as LoraPairs; None means the module is not adapted. Clone because a rotation copies the new active slot’s layers out of the pool set before re-installing them.

LoraWeights

2026-09-25: The loaded adapter set: one fixed-address pool of max_loras equal-size slots padded to max_rank, one AdapterSlot per pool slot, and per-(layer, module) [max_loras] device pointer tables.

OverlayRaw

2026-09-25: The adapter’s overlay tensors copied to device buffers of their own, with their row counts: *_base is [*_r, h] and *_delta is [*_t, h]. *_base holds either the token_adapter base table or a modules_to_save full table.

OverlayRawSlot

2026-09-25: One adapter’s OverlayRaw and its trainable ids.

OverlayTensor

2026-09-25: A classified overlay tensor: its table and its role.

OverlayTensors

2026-09-25: One adapter’s overlay tensor names by (table, role); each field is set at most once.

SlotView

2026-09-25: A cache slot’s state as select_victim_slot reads it, built by LoraWeights::cache_slot_views.

TokenOverlaySet

2026-09-25: The overlay tables for the whole adapter pool, built from the per-slot EmbedOverlays. Model-engine keeps it only when some slot has an overlay (any_active).

Enums

AdapterAb

ExpertProj

2026-09-25: Which routed-expert projection a delta targets.

LoraModule

LoraSites

2026-10-03: Which attention LoRA sites a step of one sequence runs.

LoraTarget

2026-09-25: The decoded target of one PEFT LoRA tensor.

OverlayModule

2026-09-25: The table an overlay tensor belongs to.

OverlayTensorKind

2026-09-25: The role an overlay tensor plays in build_overlay.

VictimError

2026-09-25: Why a promotion found no victim slot.

Constants

ROWDIFF_THRESH

2026-09-25: A base row whose largest absolute difference from the served row exceeds this is overridden (embed_rowdiff_bf16).

Functions

adapter_id_hash

2026-09-25: The adapter’s cache-identity key, which keeps prefix and KV reuse within one adapter. It is FNV-1a over the adapter’s name, then over generation’s little-endian bytes when generation != 0. It never depends on the pool slot, so an adapter moved to another slot keeps its id. A swap into a slot bumps that slot’s generation, which changes the id. 0 is reserved for the base model, so a hash of 0 becomes 1.

allow_partial_targets

2026-09-25: METRALE_LORA_ALLOW_PARTIAL (1 or true, any case): load an adapter that names target modules the engine cannot apply, skipping those. Re-exported from metrale-config, whose parse-time check reads the same switch, so both checks see one value. 2026-09-26: METRALE_LORA_ALLOW_PARTIAL (1 or true, any case), read once per process: load an adapter that names target modules the engine cannot apply, skipping those.

apply_expert_lora_sorted

2026-09-25: Fold proj’s per-expert deltas onto the expert-sorted grouped-GEMM output. x is the sorted input ([total_expanded, k_in]), base_out the sorted output ([total_expanded, n_out], in place), and expert_offsets_host a host copy of expert_offsets. Only experts with a pair for proj and at least one row are folded, each over its own row block.

apply_router_lora

2026-09-25: Fold the router (mlp.gate) delta onto gate_logits ([n, num_experts], in place) from router_in ([n, hidden]), in blocks of at most max_rows rows.

build_moe_row_adapter_decode

2026-09-25: The [padded_n] per-row adapter map of a decode batch, one token per sequence, which moe_lora_gather_bgmv.cu reads as row_adapter[row / top_k] and skips when negative. Row i is active when resolve_moe_lora_route gives Fold, else -1; so Skip, Refuse and padding rows (i >= adapter_slots.len()) are all -1, which is not the attention seq_slot convention (there -1 means the active adapter). Model-engine uploads it in upload_moe_row_adapter.

build_moe_row_adapter_host

2026-09-25: The [total_tokens] per-row adapter map of a packed prefill batch. cu_seqlens_host is the [batch + 1] prefix sum of per-stream token counts; stream b’s adapter_slots[b] is written, unchanged, over rows cu_seqlens_host[b]..cu_seqlens_host[b + 1]. Only tests call it today.

build_overlay

2026-09-25: Build one slot’s EmbedOverlay: clamp the trainable ids, diff and compact the embed rows, build the slot_map, and, for an untied head (!tied) with its own base, the lm_head rows. The staged buffers are freed when an overlay is returned. Ok(None) when the slot has no embed base or overrides no row. An error when the overlay kernels are not loaded. Model-engine passes tied when the lm_head shares the embed buffer or is served quantized; TokenOverlaySet::from_slots then points the lm_head at the embed rows.

build_override_set

2026-09-25: The vocab rows the overlay replaces: the indices i with row_diff[i] set, plus trainable, sorted ascending without duplicates.

build_seq_slot_host

2026-09-25: The host seq_slot[padded_n] buffer the bgmv reads, one entry per row: row i < n: adapter_slots[i] if >= 0, else active (a negative slot means the active adapter); row i >= n (padding): -1, which lora_bgmv.cu treats as no delta.

clamp_trainable_to_vocab

2026-09-25: Split trainable ids at the served vocab, keeping list order, since delta row k belongs to the k-th id. Returns (kept_ids, skipped_count):

classify_key

2026-09-25: PEFT tensor name → (layer, target, A or B). Every name it cannot place is an error naming the reason (REJECT[...]), never a skip. Whatever precedes .layers. is ignored, so model.layers.{i}.* and model.language_model.layers.{i}.* both parse; the layer index follows .layers..

classify_overlay_key

2026-09-25: Classify a PEFT tensor name as an overlay tensor. None for a name without the base_model.model. prefix, a lora_A/lora_B weight (for super::classify_key), or a name matching no overlay form. Only the module name and suffix are matched, so model.embed_tokens and model.language_model.embed_tokens both work. The token_adapter and lora_embedding suffixes are tested before the bare .weight form, so …token_adapter.base_layer.weight is not taken for a full table.

ensure_decode_route_servable

2026-09-25: An error when route is Refuse. The batched decode entries (model-engine decode_batch_compute_main, mixed_forward) call it before any graph lookup: build_moe_row_adapter_decode writes a Refuse row as -1, so the batch would otherwise run that request without its adapter.

expert_delta_workitems

2026-09-25: Work items for the adapted experts, from the expert_offsets prefix sum ([num_experts + 1]; expert e’s sorted rows are expert_offsets[e]..expert_offsets[e + 1]). An expert index past the table, or with end <= start, is skipped.

expert_router_bytes

2026-09-25: Padded bytes of the expert/router pool for the given audited keys: (stride * in + out * stride) * 2 per expert key and per router layer. stride is expert_pack::packed_stride(max_rank), the stride expert_pack::pack_into advances by, so the size matches what is packed.

full_attention_layers

is_gdn_key

2026-09-25: Whether key names a linear_attn.* tensor other than linear_attn.out_proj, the family classify_key refuses with REJECT[gdn-target]. audit_adapter skips these under METRALE_LORA_ALLOW_PARTIAL instead of matching on the refusal message.

load_lora_adapters_generic

2026-09-25: load_lora_adapters_multi with one adapter, whose slot name is empty.

load_lora_adapters_multi

2026-09-25: Load startup adapters: audit each one, check free memory, pack adapter k into slot k, then build the per-module [max_loras] pointer tables (0 where a slot has no pair) and the [max_loras] scale table. Slots past the adapter count are empty placeholders.

lora_eager_env

2026-09-25: METRALE_LORA_EAGER (1 or true, any case), from the process-wide ModelLevers::get(). The model reads its own copy, levers.lora_eager, and runs decode and verify without CUDA graphs while an adapter is loaded.

lora_experts_env

2026-09-25: METRALE_LORA_EXPERTS (1 or true, any case), read once: load routed-expert and router LoRA deltas. Unset, an adapter with expert or router tensors is refused at load (expert_pack.rs).

lora_peer_env

2026-09-25: METRALE_LORA_PEER, the host:port of the weight peer that stages adapters, when set and non-empty. Setting it also sets the model’s lora_rotatable (see lora_rotate_env).

lora_rotate_env

2026-09-25: METRALE_LORA_ROTATE (1 or true, any case), from the process-wide ModelLevers::get(). It, or lora_peer_env, sets the model’s lora_rotatable, which permits runtime adapter rotation and swap whatever the number of resident adapters. It does not make decode eager; only METRALE_LORA_EAGER does. met serve refuses --lora-stageable-disk unless this or a peer is set.

max_lora_expert_rank

2026-09-25: The padded rank of the expert and router LoRA pools, METRALE_LORA_EXPERT_RANK (a positive integer; otherwise 16), read once. It is separate from --max-lora-rank, which sizes the slot pool. An adapter whose r exceeds it is refused at load (expert_pack.rs).

no_batch_verify

2026-09-25: METRALE_LORA_NO_BATCH_VERIFY=1, read once: while an adapter is loaded, cross-sequence batched verify is not eligible (model-engine trait_impl/verify_e.rs).

override_source

2026-09-25: Where an overridden id’s row comes from: Some(k), delta row k, when id is the k-th trainable id (this wins over a differing base row); None, the adapter’s base row id.

pack_store_into_slot

2026-09-25: Disk swap: audit store and pack it into existing pool slot of lw in place, with the same audit_adapter and pack_slot as the startup load. The slot region is zeroed first, since it still holds the previous adapter. Then the slot’s name, config and pairs are replaced, its pointer and scale table entries refreshed, and its generation bumped. Returns the new pairs so the caller can re-install them when the slot is active.

prefill_bgmv_forced

2026-09-25: METRALE_LORA_PREFILL_BGMV=1, read once: the attention QKV prefill applies LoRA with the per-row apply_lora_bgmv instead of apply_lora_delta when the call has a slot buffer and a route (layers/qwen3_attention/prefill/paged_qkv.rs).

reject_pending_overlay

2026-09-25: An error when the adapter has a lora_embedding_A/B tensor; every other overlay tensor is loaded.

resolve_moe_lora_route

2026-09-25: The MoE LoRA fold decision for one request:

routed_prefill_slot_of

2026-09-25: The predicate behind LoraWeights::routed_prefill_slot. Resolves adapter_slot (>= 0 is that slot, negative is active) and returns it only when it is not active and is < num_slots; otherwise None.

router_dims

2026-09-25: (out_dim, in_dim) of the router (mlp.gate) projection. Its delta is added to the routing logits before top-k.

select_routed_pair

2026-09-25: The LoraPair for (global layer, module) in a slot’s global-layer-indexed layers. None when the index is out of range, the layer has no pairs, or the adapter has no pair for that module.

select_victim_slot

2026-09-25: Pick the slot a promotion overwrites, from cache, the (slot_index, view) pairs the caller passes (model-engine passes cache_slot_views(), slots [pinned, max_loras)):

stage_overlay_raw

2026-09-25: Copy one adapter’s overlay tensors to device buffers of their own. Ok(None) when it has none, or has a lora_embedding tensor (the audit refuses those). A modules_to_save table goes into *_base when there is no token_adapter base. An error when neither table has a base.

uniform_seq_slots

2026-10-03: The count-row slot buffer a step of one sequence uploads, or None for the installed-pair path: only a request that resolves to the active adapter, on sites with a pair fallback, takes the pair. Multi-sequence sites always get their rows’ slots, or the attention would run without the adapter (the active-adapter MTP verify did until 2026-10-03).

validate_peft_config

2026-09-25: Adapter-config checks that need --max-lora-rank: the rank fits the pool, and every target module is one LoraModule or the router (gate). The parse-time checks are in metrale_config::parse_peft_adapter_config.