Module compiler 

2026-09-26: Grammar compiler: turns a grammar, a JSON schema or a structural tag, together with a tokenizer, into a CompiledGrammar, which holds the grammar and computes, per parser state, the token masks the matcher uses to fill the logit bitmask.

GrammarCompiler caches compiled grammars in GrammarCache, and, when caching is enabled, shares one RuleLevelCache of per-rule masks across every grammar it builds.

Owner: grammar. Invariants: none beyond the types.

Re-exports

pub use mask_snapshot::SnapshotError;

pub use mask_snapshot::SnapshotIdentity;

Modules

mask_snapshot

Structs

AdaptiveTokenMask

Preprocessed accept/reject/uncertain partition for one parser state.

CompiledGrammar

A grammar compiled against a tokenizer — the result of preprocessing performed by super::GrammarCompiler.

CompiledGrammarImpl

The inner, shared state of a CompiledGrammar. Port of CompiledGrammar::Impl.

GrammarCompiler

A tokenizer-bound grammar compiler with an internal cache.

GrammarDecomposition

The result of the WGRAMMAR static/dynamic decomposition pass over a whole grammar — one RuleDecomposition per rule, indexed by rule_id.

RuleDecomposition

The decomposition of one rule body into alternating static / dynamic segments, in body order.

RuleLevelCache

Cross-grammar adaptive-token-mask cache. Cheap to clone — the inner state is shared via Arc, matching the C++ pimpl shared_ptr.

RuleMaskKey

The structural cache key — port of RuleLevelCache::Impl::NodeKey.

Enums

CompileError

An error produced while compiling a grammar, schema or tag.

Forced

The result of a forced-token analysis over a next-token bitmask.

Segment

One decomposed span of a rule body under WGRAMMAR.

StoreType

How an AdaptiveTokenMask stores its accepted/rejected partition.

Constants

UNLIMITED_SIZE

Sentinel max_cache_memory_size meaning “no memory bound” — port of RuleLevelCache::kUnlimitedSize (static_cast<size_t>(-1)).

USE_BITSET_THRESHOLD

Above this many accepted (and rejected) indices, the partition is stored as a vocab-sized bitset instead of an index vector. Faithful to the C++ AdaptiveTokenMask::USE_BITSET_THRESHOLD.

Functions

analyze_bitmask

Analyze a fully-computed next-token bitmask for the forced-token condition.

decompose_static_regions

Run the WGRAMMAR static/dynamic decomposition over an optimized grammar.