decode_token

metrale_grammar::tokenizer::decoder

Function decode_token 

Source 
pub fn decode_token(token: &str, vocab_type: VocabType) -> Vec<u8> ⓘ
Expand description

Decode one raw vocabulary token to its byte sequence for vocab_type.