Crate metrale_gpu_runtime 

2026-09-25: GPU runtime: the GpuBackend trait with its CUDA, Metal and mock implementations, device buffers, streams, the kernel registry, the cuBLASLt, CUTLASS and FlashInfer bridges, and the process-wide SSM tail-snapshot and hermetic switches.

Owner: gpu-runtime. Invariants: none beyond the types.

Modules

buffers

2026-09-25: The per-model GPU buffer arena: every intermediate tensor of a forward pass, allocated once and reused by every step.

cublaslt

2026-09-25: cuBLASLt FFI for out[M,N] = act[M,K] @ weight[N,K]ᵀ: BF16 GEMMs with BF16 or FP32 output here, FP8 E4M3 GEMMs in fp8, and the FP8 scale-tensor layout math in scale_layout.

cuda_backend

2026-09-25: The CUDA GpuBackend: MetraleCudaBackend, the driver API declarations its submodules call, and its device-allocation ledger.

cuda_host

2026-09-25: The process-scoped CUDA context and stream (CudaHost), and release, which unloads one model’s crate::registry modules.

cutlass

2026-09-25: Host-side FFI to the CUTLASS GEMM wrappers in cuda/cutlass_*.cu: the shared extern block and workspace here; dense BF16 and NVFP4 GEMMs in gemm, grouped per-expert NVFP4 MoE GEMMs in grouped, and NVFP4 weight pack, SFB swizzle and transpose in pack.

flashinfer

2026-09-25: Host-side FFI to the FlashInfer ragged (varlen) BF16 prefill attention wrapper in cuda/flashinfer_ragged_prefill.cu: several requests’ attention in one launch, delimited by qo_indptr/kv_indptr. hd128 holds the head_dim-128 entry.

gpu

2026-09-25: The GpuBackend trait: allocation, copies, kernel launches, graphs, events and memory queries for one device.

host_heap

2026-09-25: The heap allocation behind the default GpuBackend::alloc_host_pinned and free_host_pinned. The mock allocates the same zeroed (bytes, 64) layout itself, to count calls, and frees through the default.

kernel

2026-09-25: KernelModule, a PTX module loaded through cudarc, plus a 1-D launch_config helper and launch_vector_add.

kernel_args

2026-09-25: KernelLaunch, the builder for kernel launches on any GpuBackend. Each argument is recorded as a buffer or as scalar bytes, so the Metal backend can bind buffers with setBuffer:offset:atIndex: and scalars with setBytes:length:atIndex:.

mlx_int8

2026-09-25: Launches for 8-bit affine weights in the MLX layout: load a (.weight, .scales, .biases) triplet and run the dequant, GEMV, fused GEMV and GEMM kernels of kernels/metal/common/mlx_int8_*.metal.

op_cache

2026-09-25: OpCache: per-backend memo of kernel handles, grow-only device scratch, and the once / first-N gates of diagnostics.

pinned_hosts

2026-09-25: Which host buffers are page-locked, so the CUDA backend’s copy_h2d_async can keep its promise that the caller may drop the source on return.

registry

2026-09-25: MetraleRegistry: one model’s kernel modules, loaded into the process CUDA context, with raw driver-API helpers for launches, copies and device symbols.

stream

2026-09-25: MetraleStream: a CUDA stream paired with the device it was created on.

timing

2026-09-25: L1 GPU spans: CUDA event pairs around each serve lane on every step, and around each eager kernel launch on the steps the telemetry hub arms (one in kernel_span_every), at telemetry level Kernel only.

Functions

hermetic_enabled

2026-09-25: Whether hermetic mode is on: set_hermetic(true) was called or METRALE_HERMETIC=1. Resolved on first call and cached; read by the SSM snapshot lookups in metrale-cache (snapshot.rs, snapshot_tier.rs).

set_hermetic

2026-09-25: Publish met serve --hermetic, before the first hermetic_enabled call, which caches its answer. Only true is published, so without the flag METRALE_HERMETIC=1 still decides.

set_ssm_tail_midchunk

2026-09-25: Publish --no-ssm-tail-midchunk: the serve plan passes Some(false) when the flag is given and None otherwise. None publishes nothing, so METRALE_SSM_TAIL_MIDCHUNK still decides. The cell is set once: the first published value, or the first ssm_tail_midchunk_enabled read, wins.

ssm_tail_boundary

2026-09-25: The last multiple of block_size strictly below total_tokens, or None when block_size is 0 or total_tokens <= block_size.

ssm_tail_ckpt_enabled

2026-09-25: Tail-checkpoint chunk split, on only with METRALE_SSM_TAIL_CKPT=1 (see ssm_tail_boundary). Resolved on first call and cached.

ssm_tail_midchunk_enabled

2026-09-25: Mid-chunk SSM tail capture: on unless --no-ssm-tail-midchunk or METRALE_SSM_TAIL_MIDCHUNK=0. While it is on the scheduler does not end chunks at ssm_tail_boundary, and prepare_midchunk_capture (metrale-model-engine) plans an in-pass capture there, which it does only on metrale_scale builds. Resolved on first call and cached.