Crate metrale_grammar 

Re-exports

pub use api::HfMetadata;

pub use api::detect_metadata_from_hf;

pub use tokenizer::VocabType;

pub use grammar::GrammarData;

pub use grammar::GrammarExpr;

pub use grammar::GrammarExprType;

pub use grammar::Rule;

pub use grammar::TagDispatch;

pub use schema::JsonFormat;

pub use schema::SchemaConverterOptions;

pub use schema::SchemaError;

pub use schema::deepseek_xml_tool_calling_to_ebnf;

pub use schema::json_schema_to_ebnf;

pub use schema::json_schema_to_grammar;

pub use schema::minimax_xml_tool_calling_to_ebnf;

pub use schema::qwen_xml_tool_calling_to_ebnf;

pub use structural_tag::StructuralTag;

pub use structural_tag::StructuralTagError;

pub use structural_tag::structural_tag_from_items;

pub use structural_tag::structural_tag_to_grammar;

Modules

compiler

2026-09-26: Grammar compiler: turns a grammar, a JSON schema or a structural tag, together with a tokenizer, into a CompiledGrammar, which holds the grammar and computes, per parser state, the token masks the matcher uses to fill the logit bitmask.

earley

2026-09-26: Earley parser for grammar matching. EarleyParser holds the Earley items, advances them one byte at a time (advance), expands rule references (predict, predict_fsm), completes finished rules (complete), reports the acceptable next bytes (accept), and pushes or rolls back steps for the matcher.

fsm

grammar

2026-09-26: The grammar representation and its front end: the expression and rule data (GrammarExpr, GrammarData), GrammarBuilder, the EBNF lexer and parser (parse_ebnf), the normalising and optimising passes (functor), and the EBNF printer.

matcher

regex

schema

2026-09-26: JSON Schema to EBNF conversion. json_schema_to_ebnf parses the schema into a JsonValue, resolves it into a SchemaSpec (parser*), and emits the EBNF rules (converter, gen_*); json_schema_to_grammar parses that EBNF into GrammarData.

structural_tag

support

2026-09-26: Support code for the grammar crate: UTF-8, Latin-1 and hex decoding (encoding), escape handling for grammar literals (escape), union and intersection of sorted integer sets (int_set), the CSR Compact2DArray, UnionFindSet, and hash-combine helpers (hash).

tokenizer

2026-09-26: Tokenizer metadata for the grammar matcher: TokenizerInfo (the decoded vocabulary), VocabType (Raw, ByteFallback, ByteLevel), decode_token and the byte-level character maps, and detect_metadata_from_hf, which reads the vocabulary type and prefix-space flag from the text of a tokenizer.json.

Structs

BatchGrammarMatcher

A batched driver over many GrammarMatchers — fills one packed bitmask tensor for a whole decode batch in parallel.

CompiledGrammar

A grammar compiled against a tokenizer — the result of preprocessing performed by super::GrammarCompiler.

DLDataType

DLPack data type descriptor (DLDataType).

DLDevice

DLPack device descriptor (DLDevice).

DLTensor

DLPack tensor view (DLTensor) — does not own memory.

Grammar

A parsed grammar, ready to be handed to crate::GrammarCompiler::compile_grammar.

GrammarCompiler

Compiler that turns grammars / schemas / structural tags into a CompiledGrammar, bound to a single tokenizer.

GrammarMatcher

Per-request grammar matcher: accepts tokens, fills the next-token bitmask, supports rollback and termination.

StructuralTagItem

A single legacy structural-tag item: begin, a JSON schema string, and end. Port of the public StructuralTagItem struct from include/xgrammar/grammar.h.

TokenizerInfo

Decoded-vocabulary metadata the grammar matcher needs.

Enums

DLDataTypeCode

DLPack data type code enum (DLDataTypeCode).

DLDeviceType

DLPack device type enum (DLDeviceType).

Functions

allocate_token_bitmask

Allocate a token bitmask: an i32 buffer of batch_size * ceil(vocab_size/32) words, initialized to all-ones (-1, i.e. every token allowed).

get_bitmask_shape

Return the shape of the bitmask: (batch_size, ceil(vocab_size/32)).

reset_token_bitmask

Reset a bitmask buffer to the full (all-allowed) mask.