Expand description
2026-09-26: ChatTokenizer: a Hugging Face tokenizer, the chat template prompts are
rendered with (minijinja, or the DeepSeek-V4 message encoder), and a streaming decoder.
Owner: server (tokenizer). Invariants: none beyond the types.
Structsยง
- Chat
Tokenizer - Streaming
Decoder - 2026-09-26:
tokenizers::DecodeStreamwith its generic parameters fixed.