Module shared_lookup 

2026-10-04: Cross-request token cache for prompt lookup. A bounded index over recent requests’ prompt and output tokens, split into scopes keyed by (model, tokenizer, LoRA adapter, tenant). A sequence whose own history has no match may copy the continuation of the same n-gram from what earlier requests of its scope held. Host memory only. The scheduler thread owns the cache, so there is no lock.

Each scope is a token ring plus a slot table, both sized once when the scope is created. A document (one request’s tokens) is appended to the ring and each of its n-grams is written to the slot its hash selects, holding the ring position of the token that followed it. A later write to the same slot replaces the entry, and the ring overwrites its oldest tokens: both are the eviction. A stale or colliding entry is harmless, because every hit is re-checked token by token against the ring before it is used.

Runs a document repeats are not stored again: while an n-gram’s existing entry already predicts the document’s next token, nothing is written. A re-sent conversation prefix therefore costs no ring space.

Owner: speculative. Invariants:

• A proposal under scope S is a slice of tokens inserted under S: the key_len ring tokens before it equal the history’s last key_len tokens, it is non-empty, and it stops at its document’s end.
• No call under one scope reads, writes or evicts another scope’s tokens, except that creating a scope when max_scopes exist drops the least recently used scope whole.
• Memory is fixed per scope at creation: at most max_scopes scopes of SharedLookupConfig::scope_bytes each, within the configured budget.
• Deterministic: proposals depend only on the order of insert and propose calls and their arguments.

Structs

ScopeKey

2026-10-04: What partitions the cache. Two sequences share drafts only when every field is equal.

SharedLookupConfig

2026-10-04: Cache settings (--prompt-lookup-shared-cache-*).

SharedTokenCache

2026-10-04: The cross-request cache: scopes created on first insert, at most max_scopes, each of fixed size.

Constants

DEDUP_RUN

2026-10-04: Duplicate n-grams in a row, while storing, after which the rest of a repeated run is skipped. Shorter repeats are stored, so a copy is not cut into many short pieces by brief coincidences.

MIN_SCOPE_TOKENS

2026-10-04: Fewest ring tokens a scope may have.