Module ngram_cache 

2026-09-25: NVMe-backed row cache for n-gram embedding tables. Rows fault from the checkpoint file into a pinned arena, and the gather kernel reads the arena by slot.

Owner: storage, n-gram row cache. Invariants:

• map[id] == s exactly when slot_row[s] == id, on every path out of resolve.
• A slot pinned since the last end_batch is never chosen as an eviction victim.

The n-gram ids are computed on the host, so resolve maps each id to its arena slot and the caller hands the gather kernel slot indices in place of ids; table_dev_va is the table it gathers from. The arena’s device address equals its host address (ExpertArena::new refuses a host where it does not), so a miss is read from NVMe straight into its slot, with no host-to-device copy. Eviction is CLOCK (second chance) over the unpinned slots.

On unix the files are opened O_DIRECT, which needs 4 KiB-aligned transfers: a miss reads the one or two blocks covering its row into an aligned bounce buffer and copies the row out. The cache holds single rows, not blocks.

Structs

NgramRowCache

2026-09-25: One table’s backing files and its resident row cache.