Crate metrale_telemetry 

2026-09-26: Engine telemetry: run metrics, kernel audit, launch trace, startup progress events, and the metal-up instrument set.

Levels

Level::Off (where a new Telemetry starts), Level::Basic and Level::Kernel.

Layers

Layer	Module	Content
L0 device	device, sampler	NVML via dlopen, sampled at 10 Hz when serving: power, energy counter, clocks, temperature, clock-event reasons, memory, PCIe, NVLink
L1 kernel	kernel	CUDA-event spans per serve lane each step; per kernel one step in N
L2 scheduler	sched	queue depths, batch shape, loop phases, blocking syncs/D2H, graph replays/captures
L3 cache	sched, prefix_cache	KV blocks, SSM slots, prefix-cache hits/misses
L4 spec	sched::SpecMatrix	verify acceptance by draft depth
L5 request	request	TTFT, TPOT, end-to-end
Energy	energy	live J/token and J/request from NVML counter deltas

Exported by export: Prometheus text, JSONL events, and OTLP behind a feature.

Owner: telemetry. Invariants: at Level::Off, every hot-path entry point of Telemetry returns after one relaxed atomic load, with no clock read and no allocation (tests/zero_cost_off.rs).

Re-exports

pub use hub::DeviceState;

pub use hub::Telemetry;

pub use hub::global;

pub use level::ConfigError;

pub use level::Level;

pub use level::TelemetryConfig;

Modules

clock

2026-09-26: The clock crate::hub::Telemetry reads, injected so a test can count its reads (tests/zero_cost_off.rs).

device

2026-09-26: L0: what the GPU reports through NVML: power, the cumulative energy counter, clocks, temperature, clock-event reasons, memory, PCIe and NVLink.

energy

2026-09-26: Energy attribution: the NVML counter’s deltas, paired with the tokens produced over the same interval.

export

2026-09-26: Exporters. Each renders a crate::snapshot::TelemetrySnapshot or the instruments behind it into a String; the server does the I/O.

hub

2026-09-26: Telemetry: one instrument set behind one level gate.

instrument

2026-09-26: Lock-free instruments: counters, gauges and power-of-two histograms. All are const-constructible, so a whole instrument set can live in a static without a lazy initialiser.

kernel

2026-09-26: L1: GPU time from CUDA event pairs: every serve lane on every step, and every eager kernel launch on one step in N.

kernel_audit

2026-09-26: The kernel-resolution audit: every lookup the CUDA backend’s GpuBackend::kernel(module, func) makes, whether it resolved, and the call site that issued it; plus the boot gate’s seal.

launch_trace

2026-09-26: Launch trace: record the kernel launches, async memsets and device-to-device copies a region enqueues, so two executions of it can be diffed op by op (end_and_diff). The verify paths trace their captured region this way to find a host value that differs between two steps. Off until begin is called.

level

2026-09-26: The telemetry Level and the validated TelemetryConfig.

prefix_cache

2026-09-26: The PrefixCache trait (KV block reuse across requests that share a prompt prefix, plus the SSM snapshot index) and the prefix-cache hit and miss counters.

progress

2026-09-26: Structured startup-progress events for the TUI.

request

2026-09-26: L5: per-request latency (TTFT, TPOT, end-to-end) and energy.

run_metrics

2026-09-26: The run mailbox: process-global counters read by the /metrics handler and the TUI, which hold no handle to the scheduler’s context, and reset_for_new_run, which starts a new run’s accounting.

sampler

2026-09-26: The device sampler thread, the I/O half of L0 and energy. It reads a DeviceSource at the configured period and hands each reading to Telemetry::absorb_device. It runs on its own thread, so a slow driver call delays the next sample, not a scheduler step.

sched

2026-09-26: L2 scheduler and L3 cache instruments (queue depths, batch shape, loop phase timings, blocking syncs and D2H copies, CUDA-graph replays and captures, KV-block and SSM-slot occupancy) and the L4 SpecMatrix.

seqcell

2026-09-26: A sequence-locked cell of W words: readers take no lock and never see a mix of two stores.

snapshot

2026-09-26: TelemetrySnapshot: the instruments’ current values as one plain struct. The /metrics telemetry section, the /v1/events stream and the OTLP body are rendered from it.

spec_confidence

2026-10-04: Draft outcomes by drafter confidence: for each draft a verify step reached (every draft before it was accepted), its top-1 log-probability bucket and whether it was accepted. met bench spec-cost fits the acceptance calibration of --spec-cost-model measured from these counts, so P(accept | bucket) is conditional on reaching the draft, which is how the planner chains it.