Trait ModelLifecycle 

pub trait ModelLifecycle {
    // Required methods
    fn bind_gpu_to_thread(&self) -> Result<()>;
    fn alloc_sequence(&self) -> Result<SequenceState>;
    fn cache_sequence(&self, seq: &SequenceState);
    fn free_sequence(&self, seq: &mut SequenceState) -> Result<()>;
    fn compact_sequence(
        &self,
        seq: &mut SequenceState,
        new_slot: usize,
    ) -> Result<()>;
    fn detach_slot_for_reuse(&self, seq: &mut SequenceState);
    fn kv_block_bytes(&self) -> usize;

    // Provided methods
    fn teardown(&mut self) -> Result<()> { ... }
    fn poll_innerq(&self) { ... }
    fn high_speed_swap_dims(&self) -> Option<ModelDims> { ... }
    fn alloc_sequence_for(&self, budget_tokens: usize) -> Result<SequenceState> { ... }
    fn save_sequence_state(
        &self,
        _seq: &SequenceState,
        _writer: &mut dyn Write,
    ) -> Result<()> { ... }
    fn restore_sequence_state(
        &self,
        _seq: &mut SequenceState,
        _num_blocks: usize,
        _reader: &mut dyn Read,
    ) -> Result<()> { ... }
    fn num_free_blocks(&self) -> usize { ... }
    fn num_total_blocks(&self) -> usize { ... }
    fn reclaim_prefix_blocks(&self, _num_blocks: usize) -> usize { ... }
    fn reclaim_prefix_blocks_for(
        &self,
        prompt: &[u32],
        adapter_slot: i32,
        blocks_needed: usize,
    ) -> PrefixReclaim { ... }
}

2026-09-26: The model’s lifecycle and memory: teardown, thread binding, sequence allocation and release, sequence swap, and the KV pool counts.

Required Methods

fn bind_gpu_to_thread(&self) -> Result<()>

2026-09-25: Bind the GPU context to the current thread. The scheduler thread and the EP worker call it before using the model.

fn alloc_sequence(&self) -> Result<SequenceState>

2026-09-25: A new sequence with its layer states.

fn cache_sequence(&self, seq: &SequenceState)

2026-09-25: Insert the sequence’s tokens (prompt and generated) into the prefix cache. Call before Self::free_sequence, while its blocks are still valid.

fn free_sequence(&self, seq: &mut SequenceState) -> Result<()>

2026-09-25: Release the sequence’s KV blocks, prefix-cache refs, SSM pool slot and LoRA slot ref.

fn compact_sequence( &self, seq: &mut SequenceState, new_slot: usize, ) -> Result<()>

2026-09-25: Move the sequence’s SSM state (h_state and conv_state of every SSM layer) to pool slot new_slot.

fn detach_slot_for_reuse(&self, seq: &mut SequenceState)

2026-09-25: Give up a retired sequence’s SSM pool slot after compact_sequence moved a surviving sequence into it: sets slot_idx to usize::MAX and empties the slot guard, so freeing or dropping this sequence does not release the slot. Call it right after that compact_sequence, before any step that can drop the sequence.

fn kv_block_bytes(&self) -> usize

2026-10-01: Bytes one block index of the paged cache holds across every layer, K and V, so num_total_blocks() * kv_block_bytes() is the pool’s size. 0 for a model without a paged cache. No default: a model that has a pool must say what its blocks cost.

Provided Methods

fn teardown(&mut self) -> Result<()>

2026-09-25: Release the device memory this model owns, newest first. The scheduler calls it once at shutdown (core/finish.rs). Default: Ok(()), for models that own no pooled device memory; a model that owns pools and keeps the default leaks them.

fn poll_innerq(&self)

2026-09-25: Poll this model’s TQ+ InnerQ calibration driver; the scheduler calls it once per prefill step. Default: no-op.

fn high_speed_swap_dims(&self) -> Option<ModelDims>

2026-09-25: Model dimensions for the --high-speed-swap orchestrator; None for a model that does not support it (the default).

fn alloc_sequence_for(&self, budget_tokens: usize) -> Result<SequenceState>

2026-09-25: Self::alloc_sequence for a request that can reach at most budget_tokens (prompt_len + max_tokens), so proposer state that scales with context can be sized to that (DraftProposer::alloc_state_for). Default: Self::alloc_sequence.

fn save_sequence_state( &self, _seq: &SequenceState, _writer: &mut dyn Write, ) -> Result<()>

2026-09-25: Write the sequence’s KV blocks and SSM state to writer, in a format the model owns, without freeing anything. Default: an error.

fn restore_sequence_state( &self, _seq: &mut SequenceState, _num_blocks: usize, _reader: &mut dyn Read, ) -> Result<()>

2026-09-25: Read state written by Self::save_sequence_state into an allocated sequence, allocating num_blocks KV blocks. Default: an error.

fn num_free_blocks(&self) -> usize

2026-09-25: Free KV blocks. Default 0.

fn num_total_blocks(&self) -> usize

2026-09-25: Total KV blocks in the paged cache. Default 0.

fn reclaim_prefix_blocks(&self, _num_blocks: usize) -> usize

2026-09-25: Evict up to num_blocks blocks from the prefix cache; returns how many became free. Swap-in and preemption resume check num_free_blocks() before they allocate, so they call this to reach blocks the cache holds. 0 means nothing was evictable, and the caller stops asking. Default 0.

fn reclaim_prefix_blocks_for( &self, prompt: &[u32], adapter_slot: i32, blocks_needed: usize, ) -> PrefixReclaim

2026-09-30: Reclaim prefix-cache blocks until num_free_blocks() covers what a prefill of prompt under adapter_slot allocates: blocks_needed (the caller’s count for the whole prompt, headroom included) less the blocks of the cached prefix that prefill will reuse. An empty prompt means the prefill reuses nothing (prompt logprobs). Admission and preemption resume call it before they allocate. Default: no prefix is reused, and Self::reclaim_prefix_blocks runs until blocks_needed is free or it returns 0.

Implementors

impl ModelLifecycle for NllbGpuModel

impl ModelLifecycle for TransformerModel