Functionsยง
- byte_
to_ char_ map - The GPT-2 bytes-to-unicode forward map: byte -> unicode codepoint.
- char_
to_ byte_ map - The inverse of the GPT-2 bytes-to-unicode map.
- decode_
token - Decode one raw vocabulary token to its byte sequence for
vocab_type.