pub fn byte_to_char_map() -> &'static [u32; 256]Expand description
The GPT-2 bytes-to-unicode forward map: byte -> unicode codepoint.
This is the table HuggingFace uses to encode the vocabulary; it is
the exact inverse of CHAR_TO_BYTE_MAP. Derived (not duplicated)
from the inverse table to keep a single source of truth.