Module deepseek_v41_ref 

2026-09-25: DeepSeek-V4.1 Flash tiny-graph reference: the V4.1 graph on the CPU, checked against ds41_tiny_golden.json.

Nothing here runs on a GPU. The GPU attention (attn_v41) uses its RoPE tables, window indices and top-k selections, and model::forward is the oracle the GPU tests compare against.

Where the truth comes from

ds41_tiny_golden.json is produced by gen_ds41_tiny_golden.py (beside it), which runs DeepSeek’s inference/model.py (the revision the golden’s fixture.reference records) on a tiny synthetic model: hc_mult=4 hyper-connections with delayed mixes, engram on two layers with the reference’s hash (engram.py), shared compressed attention with two kv sources, three index sources and a candidate prefilter whose consumers share its ratio, sqrt-softplus routing with a correction bias, route_scale, the swiglu clamp and a shared expert. The tilelang kernels the reference imports are replaced by ds41_ref_shims.py (pure torch).

Determinism, and why only outputs are committed

Weights and inputs are RNG-free and order-free: fixed_value(name, i, scale, offset) here reproduces the generator’s filler (fnv1a64 salt, splitmix64, 24-bit uniform, f64 arithmetic, f32 result, bf16 RNE where the parameter is stored bf16). So the golden carries only outputs: each capture as {shape, n, stride, ck, data} where data is a prime-strided sample and ck is the fp64 index-weighted checksum over the whole tensor. tests.rs checks the regeneration against the golden.

The engram hash parameters come from the GGUF

Measured 2026-09-15 with gen_ds41_engram_hash_check.py: the token map, bucket primes, slot offsets and hash multipliers that engram.py computes from the V4.1 tokenizer equal the deepseek41.engram.* metadata of the Q2_K GGUF, so the loader builds the engram tables from that metadata alone (EngramHashTables::from_flat).

Owner: model-arch, DeepSeek-V4.1 reference. Invariants: none beyond the types.

Modules

attn

2026-09-25: DeepSeek-V4.1 attention for ratio-0 layers (the reference’s Attention.forward at compress_ratio == 0), CPU reference against the golden: low-rank Q with RMSNorm, one latent KV row per token (wkv + kv_norm), RoPE on the last rope_dim of both, the KV row rounded through fp8 in place (block 32, power-of-two scale), a sliding-window ring cache, sparse_attn with the attention sink, the inverse rotation on the output, then the grouped low-rank output projection (wo_a per group, then wo_b). Also the RoPE, window and fp8 helpers the compressed layers and attn_v41 share.

compress

2026-09-25: DeepSeek-V4.1 compressed attention (compress_ratio > 0), CPU reference against the golden: YaRN RoPE with compress_rope_theta, the two fp4 quantisers, linear_f32, and attention_any, the attention forward for any ratio. The compressor and the indexer are in the compressor and indexer submodules; the ratio-0 pieces are shared with attn.rs.

engram

2026-09-25: DeepSeek-V4.1 engram, CPU reference, checked stage by stage against the golden:

hc

2026-09-25: DeepSeek-V4.1 block scaffold: hyper-connections (hc_mixes with the Sinkhorn split, hc_pre, hc_post) and RMSNorm, CPU reference against the golden.

model

2026-09-25: DeepSeek-V4.1 full tiny-graph model: the config, weights, state and trace types that forward (in model/forward.rs) fills. On the fixture: embed, expand to hc_mult copies, the one-hot initial pre-mix, engram before layers 1 and 4, six blocks with the delayed mixes, collapse, final RMSNorm, f32 logits. The per-layer caches and the shared slots carry across prefill and decode, so the golden’s captures for every layer and the logits are checkable.

moe

2026-09-25: DeepSeek-V4.1 MoE (the reference’s Gate, Expert, MoE), CPU reference against the golden: sqrt-softplus routing with a correction bias that picks experts but does not scale them, top-k renormalisation, route_scale, the SwiGLU clamp, and one shared expert every token goes through.

Structs

Golden

GoldenTensor

2026-09-25: One committed capture.

WeightMeta

2026-09-25: One parameter’s regeneration rule, from weights_meta.

Constants

GOLDEN_JSON

2026-09-25: The tracked golden, produced by gen_ds41_tiny_golden.py.

Functions

assert_close

2026-09-25: Elementwise max-abs comparison that names the worst index and both values.

checksum

2026-09-25: fp64 index-weighted checksum sum(v[i] * (i + 1)), the generator’s ck. Summed sequentially here and by a torch reduction there, so callers compare with a relative tolerance.

fixed_int

2026-09-25: The generator’s integer filler (fixed_ints): fixed_raw % modulus.

fixed_raw

2026-09-25: Raw stream value for element i of the tensor named name.

fixed_value

2026-09-25: The generator’s filler: u = (z >> 40) / 2^24, f32((2u - 1) * scale + offset) in f64.

fnv1a64

2026-09-25: FNV-1a 64 over the UTF-8 bytes of s; the per-tensor salt.

regen_param

2026-09-25: Regenerate an f32-kind parameter (bf16- or f32-stored) as the generator initialised it: the rule (scale, offset) and the storage dtype come from weights_meta. Panics on another kind; the fp8 tables are regenerated by engram::regen_table.

splitmix64

2026-09-25: The splitmix64 finaliser of gen_ds41_tiny_golden.py::splitmix64.

to_bf16_rne

2026-09-25: Round an f32 to bf16 precision, round-to-nearest-even; NaN passes through.