Struct OpCache 

pub struct OpCache { /* private fields */ }

2026-09-25: Memoized kernel handles and scratch allocations for one backend.

Implementations

impl OpCache

pub fn new() -> Self

pub fn record_n_tile(&self, kernel: KernelHandle, n_tile: u32)

2026-09-29: Record the N tile kernel publishes; a later value for the same handle replaces it.

pub fn n_tile(&self, kernel: KernelHandle) -> Option<u32>

2026-09-29: The N tile recorded for kernel, or None when it publishes none.

pub fn record_a_e4m3(&self, kernel: KernelHandle)

2026-09-29: Record that kernel casts its A operand to E4M3 with no scale.

pub fn a_e4m3(&self, kernel: KernelHandle) -> bool

2026-09-29: Whether kernel was recorded as casting its A operand to E4M3.

pub fn kernel( &self, gpu: &dyn GpuBackend, module: &'static str, func: &'static str, ) -> Result<KernelHandle>

2026-09-25: Resolve module::func, memoized: gpu.kernel(..) on a miss, a map read on a hit.

pub fn scratch( &self, gpu: &dyn GpuBackend, tag: &'static str, bytes: usize, ) -> Result<DevicePtr>

2026-09-25: A scratch allocation of at least bytes, memoized under tag.

Grow-only: a larger request allocates a new buffer and drops the old pointer from the cache without freeing it. On the CUDA backend the old buffer stays on the allocation ledger until sweep_unreleased frees it when the backend drops.

pub fn alloc_fell_back(&self) -> bool

2026-09-25: Whether a loader latched the managed-memory fallback on this backend. The latch is per backend, so the next model’s load tries device memory again.

pub fn note_alloc_fallback(&self)

2026-09-25: Latch the managed-memory fallback for the rest of this backend’s life.

pub fn first_n(&self, key: &'static str, n: u32) -> bool

2026-09-25: true for the first n times this backend reaches key. Counted per backend, so a second model reports its own.

pub fn once(&self, key: &'static str) -> bool

2026-09-25: true the first time this backend reaches key, false after. Shares first_shape’s set, as (key, 0, 0, 0).

pub fn first_shape(&self, name: &str, m: u32, n: u32, k: u32) -> bool

2026-09-25: true the first time this backend sees (name, m, n, k); used to de-duplicate the GEMM route and shape log lines.

Trait Implementations

impl Debug for OpCache

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for OpCache

fn default() -> OpCache

Returns the “default value” for a type.

impl ModelResource<dyn GpuBackend> for OpCache

2026-09-25: Frees every scratch allocation and empties the scratch map.

Kernel handles need no release here: they point into the modules of the backend’s MetraleRegistry, which unloads them when it is dropped or passed to cuda_host::release.

fn label(&self) -> &'static str

2026-09-25: Human name, used to attribute a failure in Teardown::release_all’s error.

fn release(&mut self, gpu: &dyn GpuBackend) -> Result<()>

2026-09-25: Release everything this owns. Must be idempotent.