Struct TokenBitmask 

pub struct TokenBitmask { /* private fields */ }

A packed bit-per-token accept mask owned by a super::GrammarMatcher.

This is the Rust equivalent of the C++ DLTensor int32 bitmask. It is vocab_size bits wide, stored LSB-first in i32 words. The matcher fills it in place each decode step; callers read which tokens are legal via Self::is_set or apply it to logits.

Implementations

impl TokenBitmask

pub fn new(vocab_size: usize) -> Self

Allocate an all-rejecting (all-zero) mask for vocab_size tokens.

pub fn vocab_size(&self) -> usize

The vocabulary size this mask was sized for.

pub fn as_words(&self) -> &[i32]

The packed i32 words — the layout Metrale Engine hands to the kernel.

pub fn as_words_mut(&mut self) -> &mut [i32]

Mutable access to the packed words (for the W7 shim / kernels).

pub fn is_set(&self, token: usize) -> bool

True if token id token is accepted (its bit is set).

Panics if token >= vocab_size.

pub fn set(&mut self, token: usize, value: bool)

Set / clear token id token. Panics if token >= vocab_size.

pub fn clear(&mut self)

Reject every token (clear all bits).

pub fn fill_all(&mut self)

Accept every token (set every logical bit).

pub fn count_set(&self) -> usize

Number of accepted (set) tokens over the logical vocabulary.

pub fn all_set(&self) -> bool

True if every logical token bit is set.

pub fn rejected_tokens(&self) -> Vec<i32>

Collect the rejected (cleared) token ids — the _DebugGetMaskedTokensFromBitmask helper of the C++ matcher.

Trait Implementations

impl Clone for TokenBitmask

fn clone(&self) -> TokenBitmask

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenBitmask

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'a> From<&'a mut TokenBitmask> for BitmaskSlice<'a>

fn from(b: &'a mut TokenBitmask) -> Self

Converts to this type from the input type.

impl PartialEq for TokenBitmask

fn eq(&self, other: &TokenBitmask) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Eq for TokenBitmask

impl StructuralPartialEq for TokenBitmask