pub const USE_BITSET_THRESHOLD: usize = 1000;Expand description
Above this many accepted (and rejected) indices, the partition is
stored as a vocab-sized bitset instead of an index vector. Faithful
to the C++ AdaptiveTokenMask::USE_BITSET_THRESHOLD.