bitmask_size

Function bitmask_size 

Source
pub fn bitmask_size(vocab_size: usize) -> usize
Expand description

Number of i32 words needed to hold a bit-per-token mask for a vocabulary of vocab_size tokens.

Port of GetBitmaskSize / DynamicBitset::GetBufferSize: ceil(vocab_size / 32).