Module hc 

2026-09-25: DeepSeek-V4.1 block scaffold: hyper-connections (hc_mixes with the Sinkhorn split, hc_pre, hc_post) and RMSNorm, CPU reference against the golden.

The residual stream is hc_mult parallel copies [tokens, hc, dim]. A sub-layer sits between hc_pre (collapse the copies to one input, weighted by a pre mix) and hc_post (expand its output back out, mixing the residual in through comb). hc_mixes derives all three coefficient sets from the stream itself, one projection of the flattened hc * dim stream scaled by its reciprocal RMS, then split by split_sinkhorn: pre = sigmoid(.) + eps, post = 2 sigmoid(.), comb = row softmax + eps, one column normalisation, then iters - 1 row-then-column rounds.

The mixes are delayed: a block’s attention uses the pre its predecessor’s FFN produced, and its FFN uses the pre its own attention produced. The tests reproduce that order and check every intermediate the golden captured.

Orientation of comb: with comb[j][k], the output copy is k and the residual copies j are summed, y[k] = post[k] * x + sum_j comb[j][k] * residual[j].

Owner: model-arch, DeepSeek-V4.1 reference. Invariants: none beyond the types.

Functions

hc_mixes

2026-09-25: The site mixes. x: [tokens, hc, dim] (bf16 values); hc_fn: [(2+hc)*hc, hc*dim] f32. Returns (pre[tokens, hc], post[tokens, hc], comb[tokens, hc, hc]).

hc_post

2026-09-25: y[k][d] = post[k] * x[d] + sum_j comb[j][k] * residual[j][d], to bf16.

hc_pre

2026-09-25: Collapse the copies, y[d] = sum_c pre[c] * x[c][d], back to bf16.

rms_norm

2026-09-25: RMSNorm: w * (x * rsqrt(mean(x^2) + eps)) in f32, back to bf16.

split_sinkhorn

2026-09-25: ds41_ref_shims.hc_split_sinkhorn for one token. mixes: [(2 + hc) * hc]. Returns (pre[hc], post[hc], comb[hc * hc] row-major [j][k]), all f32.