Struct AdaptiveTokenMask 

pub struct AdaptiveTokenMask {
    pub store_type: StoreType,
    pub accepted_indices: Vec<i32>,
    pub rejected_indices: Vec<i32>,
    pub accepted_bitset: BitVec,
    pub uncertain_indices: Vec<i32>,
}

Preprocessed accept/reject/uncertain partition for one parser state.

Fields

store_type: StoreType

Which representation the partition uses.

accepted_indices: Vec<i32>

Accepted sorted-vocab indices (only for StoreType::Accepted).

rejected_indices: Vec<i32>

Rejected sorted-vocab indices (only for StoreType::Rejected).

accepted_bitset: BitVec

Accepted token ids as a bitset (only for StoreType::AcceptedBitset); length == vocab size.

uncertain_indices: Vec<i32>

Uncertain sorted-vocab indices (always stored directly).

Implementations

impl AdaptiveTokenMask

pub fn from_accepted_rejected( vocab_size: usize, sorted_decoded_vocab: &[(i32, Vec<u8>)], accepted_indices: &[i32], rejected_indices: &[i32], uncertain_indices: &[i32], ) -> Self

Build a mask from accepted + rejected + uncertain partitions.

Port of the four-argument C++ constructor. sorted_decoded_vocab supplies the index -> token_id mapping for the bitset form.

pub fn from_accepted( vocab_size: usize, sorted_decoded_vocab: &[(i32, Vec<u8>)], accepted_indices: &[i32], uncertain_indices: &[i32], ) -> Self

Build a mask from only accepted + uncertain partitions (the rejected partition was not materialized — too large to be worth storing). Port of the three-argument C++ constructor.

pub fn empty(store_type: StoreType, vocab_size: usize) -> Self

An empty mask of the given store type — used internally and for the vocab_size == 0 degenerate compile path.

pub fn materialize( &self, sorted_decoded_vocab: &[(i32, Vec<u8>)], ) -> (Vec<i32>, Vec<i32>)

Materialize the accepted / rejected partitions as explicit sorted-vocab index vectors, regardless of StoreType.

Port of the partition-reconstruction logic in AdaptiveTokenMask::Print. Returns (accepted, rejected).

pub fn memory_size(&self) -> usize

Approximate heap memory used by this mask, in bytes. Port of the C++ MemorySize(const AdaptiveTokenMask&) friend.

Trait Implementations

impl Clone for AdaptiveTokenMask

fn clone(&self) -> AdaptiveTokenMask

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for AdaptiveTokenMask

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for AdaptiveTokenMask

fn eq(&self, other: &AdaptiveTokenMask) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Eq for AdaptiveTokenMask

impl StructuralPartialEq for AdaptiveTokenMask