Expand description
2026-09-26: Tokenizer metadata for the grammar matcher: TokenizerInfo (the decoded
vocabulary), VocabType (Raw, ByteFallback, ByteLevel), decode_token and the
byte-level character maps, and detect_metadata_from_hf, which reads the vocabulary
type and prefix-space flag from the text of a tokenizer.json.
Owner: grammar. Invariants: none beyond the types.
Re-exports§
pub use decoder::byte_to_char_map;pub use decoder::char_to_byte_map;pub use decoder::decode_token;pub use hf_metadata::HfMetadata;pub use hf_metadata::detect_metadata_from_hf;pub use info::TokenizerInfo;pub use vocab_type::VocabType;