DraftRunner

metrale_model_layers::circuit_exec::program

Trait DraftRunner 

Source 
pub trait DraftRunner: Send + Sync {
    // Required methods
    fn serves(&self, rows: u64) -> bool;
    fn run_draft(
        &self,
        gpu: &dyn GpuBackend,
        stream: u64,
        rows: u64,
        max_blocks_per_seq: u32,
    ) -> Result<()>;
}
Expand description

2026-09-29: A program another component runs in place of its own forward (the MTP draft head’s step), without GDN state. 2026-09-30: One program per row count: one row for a single sequence’s draft, n rows for the batched propose’s.

Required Methods§

Source

fn serves(&self, rows: u64) -> bool

2026-09-30: Whether a program for rows draft rows was compiled.

Source

fn run_draft( &self, gpu: &dyn GpuBackend, stream: u64, rows: u64, max_blocks_per_seq: u32, ) -> Result<()>

2026-09-29: Issue the rows-row program on stream for a step whose block table is max_blocks_per_seq wide; an error when none was compiled for rows.

Implementors§