pub struct TokenOverlaySet {
pub overlays: Vec<Option<EmbedOverlay>>,
pub embed_slot_map_table: DevicePtr,
pub embed_rows_table: DevicePtr,
pub embed_n_table: DevicePtr,
pub vocab: u32,
pub lmhead_rows_table: DevicePtr,
pub lmhead_ids_table: DevicePtr,
pub n_override_table: DevicePtr,
pub max_n_override: u32,
}Expand description
2026-09-25: The overlay tables for the whole adapter pool, built from the
per-slot EmbedOverlays. Model-engine keeps it only when some slot has an
overlay (any_active).
Fields§
§overlays: Vec<Option<EmbedOverlay>>2026-09-25: The per-slot overlays, kept so the buffers the tables point into stay allocated.
embed_slot_map_table: DevicePtr2026-09-25: u64[max_loras] of i32[vocab] slot_map pointers, 0 for
a slot without an overlay.
embed_rows_table: DevicePtr2026-09-25: u64[max_loras] of embed row pointers ([n, h] BF16).
embed_n_table: DevicePtr2026-09-25: u32[max_loras] embed row counts; the embed kernel skips a
slot_map entry not below it. n_override_table is the lm_head count.
vocab: u322026-09-25: The slot_map length of every overlay (0 when there is
none); the kernels skip token ids >= vocab.
lmhead_rows_table: DevicePtr2026-09-25: u64[max_loras] of lm_head row pointers.
lmhead_ids_table: DevicePtr2026-09-25: u64[max_loras] of lm_head id pointers (u32[n]).
n_override_table: DevicePtr2026-09-25: u32[max_loras] lm_head row counts; 0 skips the slot.
max_n_override: u322026-09-25: The largest lm_head row count, the lm_head kernel’s
grid.y (token_overlay.cu).
Implementations§
Source§impl TokenOverlaySet
impl TokenOverlaySet
Sourcepub fn from_slots(
gpu: &dyn GpuBackend,
overlays: Vec<Option<EmbedOverlay>>,
max_loras: usize,
tied: bool,
) -> Result<Self>
pub fn from_slots( gpu: &dyn GpuBackend, overlays: Vec<Option<EmbedOverlay>>, max_loras: usize, tied: bool, ) -> Result<Self>
2026-09-25: Build the tables. A slot’s lm_head cells are its own lm_head
overlay when it has one; else, when tied, its embed rows and ids;
else empty (count 0). An error when two overlays were built against
different vocab sizes.
Sourcepub fn any_active(&self) -> bool
pub fn any_active(&self) -> bool
2026-09-25: True when at least one slot has an overlay.