Struct ChatTokenizer 

pub struct ChatTokenizer { /* private fields */ }

Implementations

impl ChatTokenizer

pub fn image_pad_token_id(&self, declared: u32) -> Option<u32>

2026-09-26: The image placeholder token id.

declared is the image_token_id from the checkpoint’s config (top level or under vision_config) and wins when non-zero. Otherwise the id is that of <|image_pad|> when the string encodes to exactly one token, else None. With None, expand_vision_pads leaves image placeholders unexpanded and reports nothing.

pub fn video_pad_token_id(&self, declared: u32) -> Option<u32>

2026-09-26: The video placeholder token id, with the same rule as image_pad_token_id: declared when non-zero, else <|video_pad|> when it encodes to one token, else None.

pub fn expand_vision_pads( &self, tokens: Vec<u32>, pad_counts: &[usize], declared: (u32, u32), ) -> Vec<u32>

2026-09-26: Expand the image and video placeholders of a rendered token sequence, the i-th placeholder to pad_counts[i] copies (at least one), so the prompt holds one placeholder per vision-encoder output row. The Qwen3.6 templates in test_data/chat_templates write one placeholder per image or video. Returns tokens unchanged when every count is at most 1; see fan_out_pads for mismatched counts.

impl ChatTokenizer

pub fn from_model_dir( model_dir: &Path, eos_token_id: u32, supports_thinking: bool, model_type: &str, repo_root: Option<&Path>, disable_template_overrides: bool, ) -> Result<Self>

pub fn inner(&self) -> &Tokenizer

2026-09-26: The underlying Hugging Face tokenizer.

pub fn encode(&self, text: &str) -> Result<Vec<u32>>

pub fn decode(&self, ids: &[u32]) -> Result<String>

pub fn decode_with_special(&self, ids: &[u32]) -> Result<String>

2026-09-26: Decode without skipping special tokens; decode skips them.

pub fn incremental_decode( &self, toks: &[u32], prefix_offset: &mut usize, read_offset: &mut usize, ) -> String

2026-09-26: Incremental detokenizer. Returns the text toks added since the last call and advances the offsets.

Each call decodes only toks[prefix_offset..read_offset] and toks[prefix_offset..] (with decode, so special tokens are skipped) and returns what the second adds to the first. It returns an empty string and leaves the offsets alone when the second decode is not longer, ends in U+FFFD, or does not split at a char boundary. Offsets that no longer fit toks are reset to 0 first.

pub fn streaming_decoder( &self, skip_special_tokens: bool, ) -> StreamingDecoder<'_>

2026-09-26: A new StreamingDecoder over this tokenizer.

pub fn apply_chat_template_jinja( &self, messages: &[Value], tools: Option<&[Value]>, enable_thinking: bool, disable_tool_steering: bool, ) -> Result<Vec<u32>>

2026-09-26: apply_chat_template_jinja_with_effort with no reasoning effort and preserve_thinking unset.

pub fn apply_chat_template_jinja_with_effort( &self, messages: &[Value], tools: Option<&[Value]>, enable_thinking: bool, disable_tool_steering: bool, reasoning_effort: Option<&str>, preserve_thinking: Option<bool>, ) -> Result<Vec<u32>>

pub fn apply_chat_template_openai( &self, messages: &[Value], tools: Option<&[Value]>, enable_thinking: bool, disable_tool_steering: bool, ) -> Result<Vec<u32>>

2026-09-26: apply_chat_template_openai_with_effort with no reasoning effort and preserve_thinking unset.

pub fn apply_chat_template_openai_with_effort( &self, messages: &[Value], tools: Option<&[Value]>, enable_thinking: bool, disable_tool_steering: bool, reasoning_effort: Option<&str>, preserve_thinking: Option<bool>, ) -> Result<Vec<u32>>

pub fn apply_chat_template( &self, messages: &[(String, String)], enable_thinking: bool, _image_pad_counts: &[usize], ) -> Result<Vec<u32>>

2026-09-26: Render (role, content) pairs through apply_chat_template_jinja, with no tools; _image_pad_counts is unused.

pub fn eos_token_id(&self) -> u32

pub fn think_end_token_id(&self) -> Option<u32>

pub fn supports_thinking(&self) -> bool

pub fn uses_deepseek_v4_encoding(&self) -> bool