Struct HighSpeedSwap

Source

pub struct HighSpeedSwap { /* private fields */ }

Implementations§

Source §

impl HighSpeedSwap

Source

pub fn offload_block( &mut self, ctx: &CudaCtx, layer: u32, block: u32, k_block_dev: u64, k_block_host: &[bf16], v_block_host: &[bf16], ) -> Result<()>

2026-09-25: Write a KV block to the tier and project its K for the predictor. k_block_dev, k_block_host and v_block_host are BF16 [block_size, num_kv_heads, head_dim]; the device K feeds the projection, the host copies the per-kv_head groups.

Source

pub fn offload_block_on_stream( &mut self, stream: u64, layer: u32, block: u32, k_block_dev: u64, k_block_host: &[bf16], v_block_host: &[bf16], ) -> Result<()>

2026-09-25: offload_block on stream, which must belong to the current thread’s CUDA context.

Source

pub fn offload_block_no_predict_on_stream( &mut self, stream: u64, layer: u32, block: u32, k_block_host: &[bf16], v_block_host: &[bf16], ) -> Result<()>

2026-09-25: offload_block_on_stream without the predictor projection, for callers whose device K is not BF16. The host buffers are still BF16.

Source

pub fn attend_layer( &mut self, ctx: &CudaCtx, layer: u32, seq_block_ids: &[u32], q_dev: u64, output_dev: u64, ) -> Result<()>

2026-09-25: Streaming attention for one (layer, sequence). q_dev is the step’s BF16 query [num_q_heads × head_dim], seq_block_ids the sequence’s blocks, and output_dev receives the BF16 output [num_q_heads × head_dim].

Source

pub fn attend_layer_on_stream( &mut self, stream: u64, layer: u32, seq_block_ids: &[u32], q_dev: u64, output_dev: u64, ) -> Result<()>

2026-09-25: attend_layer on stream, which must belong to the current thread’s CUDA context, with no causal mask: every slot of the last block is attended. A query that must not see later tokens of its own block uses attend_layer_on_stream_with_q_pos.

Source

pub fn attend_layer_on_stream_with_q_pos( &mut self, stream: u64, layer: u32, seq_block_ids: &[u32], q_dev: u64, output_dev: u64, last_block_valid_slots: i32, ) -> Result<()>

2026-09-25: Only the first last_block_valid_slots token slots of the last block in seq_block_ids are attended; for a query at position q_pos, pass (q_pos % block_size) + 1.

Source

pub fn new( ctx: &CudaCtx, cfg: HighSpeedSwapConfig, model: ModelDims, ) -> Result<Self>

Source

pub fn new_on_stream( stream: u64, cfg: HighSpeedSwapConfig, model: ModelDims, ) -> Result<Self>

2026-09-25: Build on stream, which must belong to the current thread’s CUDA context. The stream carries only the construction-time upload of the predictor’s projection; per-step methods take their own stream.

Source