Struct TokenizerInfo 

pub struct TokenizerInfo { /* private fields */ }

Decoded-vocabulary view of a tokenizer for grammar-constrained decoding.

Construct with TokenizerInfo::new from a raw vocabulary, or with TokenizerInfo::from_huggingface to additionally infer the vocab type / prefix-space from a tokenizer.json.

Implementations

impl TokenizerInfo

pub fn new( encoded_vocab: &[String], vocab_type: VocabType, vocab_size: Option<usize>, stop_token_ids: Option<Vec<i32>>, add_prefix_space: bool, ) -> Self

Build a TokenizerInfo from a raw (encoded) vocabulary.

Mirrors the C++ TokenizerInfo constructor:

• encoded_vocab — raw vocabulary strings, indexed by token id.
• vocab_type — controls token decoding (see VocabType).
• vocab_size — total id space; None means encoded_vocab.len(). When larger, the trailing ids become padding special tokens.
• stop_token_ids — explicit stop ids; None triggers detection via DETECTION_STOP_TOKENS.
• add_prefix_space — recorded for metadata, no decoding effect.

pub fn from_huggingface( encoded_vocab: &[String], tokenizer_json: &str, vocab_size: Option<usize>, stop_token_ids: Option<Vec<i32>>, ) -> Result<Self, String>

Build a TokenizerInfo from the raw text of a HuggingFace tokenizer.json, inferring vocab_type and add_prefix_space.

Returns Err when tokenizer_json is not a parseable JSON object. The caller still supplies the raw vocabulary, optional vocab_size, and optional stop_token_ids.

pub fn vocab_type(&self) -> VocabType

The vocabulary type.

pub fn vocab_size(&self) -> usize

The total id space size (raw vocab length, or the override).

pub fn add_prefix_space(&self) -> bool

Whether the tokenizer prepends a space to the first token.

pub fn decoded_vocab(&self) -> &[Vec<u8>]

Decoded token bytes, indexed by token id (length vocab_size).

pub fn stop_token_ids(&self) -> &[i32]

Ids accepted as end-of-generation by the grammar matcher.

pub fn special_token_ids(&self) -> &[i32]

Ids masked out during grammar-guided generation.

pub fn sorted_decoded_vocab(&self) -> &[(i32, Vec<u8>)]

(id, token) pairs sorted lexicographically by token bytes, excluding stop and special tokens.

pub fn trie_subtree_nodes_range(&self) -> &[i32]

Pseudo-trie subtree end-indices over Self::sorted_decoded_vocab.

pub fn fingerprint(&self) -> u64

A PROCESS-STABLE digest of everything an adaptive token mask depends on: the vocab type / size / prefix-space flag, the sorted decoded vocabulary (ids and bytes, in order), and the stop / special id sets.

#918: the on-disk mask snapshot (crate::compiler::mask_snapshot) is only valid for the exact tokenizer it was computed against — a mask is a partition of Self::sorted_decoded_vocab by index. This is the key that decides hit vs miss across processes, so it is FNV-1a with a fixed basis rather than ahash/DefaultHasher, whose seeds are randomized per process.

pub fn dump_metadata(&self) -> String

Serialize the metadata to the compact JSON form C++ DumpMetadata emits: {"vocab_type":N,"vocab_size":N,"add_prefix_space":b,"stop_token_ids":[...]}.

Trait Implementations

impl Clone for TokenizerInfo

fn clone(&self) -> TokenizerInfo

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenizerInfo

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for TokenizerInfo

fn eq(&self, other: &TokenizerInfo) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Eq for TokenizerInfo

impl StructuralPartialEq for TokenizerInfo