Module hf_metadata

metrale_grammar::tokenizer

Module hf_metadata 

Source

Structs§

HfMetadata
Metadata inferred from a HuggingFace tokenizer.json.

Functions§

detect_metadata_from_hf
Detect HfMetadata from the raw text of a tokenizer.json.