Struct Sampler 

pub struct Sampler { /* private fields */ }

2026-09-25: Copies device logits to the host and picks tokens there.

Implementations

impl Sampler

pub fn new(vocab_size: usize) -> Self

pub fn sample( &mut self, logits_ptr: DevicePtr, params: &SamplingParams, gpu: &dyn GpuBackend, ) -> Result<u32>

2026-09-25: Pick one token from [vocab_size] BF16 logits on the device.

The pick runs on the host. Greedy params take argmax_bf16 of the raw row, with no penalty or bias. Otherwise the row is expanded to FP32 and passed to sample_with_params with an empty history.

pub fn sample_batch( &mut self, logits_ptr: DevicePtr, batch_size: usize, params: &[&SamplingParams], gpu: &dyn GpuBackend, ) -> Result<Vec<u32>>

2026-09-25: Pick one token per row of [batch_size, vocab_size] BF16 logits on the device, as sample does for one row.

Row i uses params[i], or params[0] when params is shorter; it panics when params is empty.