Expand description
2026-09-26: KAT equality: the same sample, in different request orders, byte for byte.
A known-answer test is a claim about its input only if the output does not
also depend on what ran before it. This gate issues one BFCL draw against
one server in two or more orders and requires every sample_id’s reply to
be byte-identical across them.
ssm-state-poisoning-gate cannot answer this: it replays one fixed script
in the same order every round, and it counts a reworded reply
(RoundVerdict::Jittered) as a pass. Here a reworded reply is the finding.
Owner: bench, kat_equality. Invariants: none beyond the types.
Re-exports§
pub use compare::Observation;pub use compare::OrderRun;pub use compare::SampleVerdict;pub use compare::Score;pub use compare::permutation;pub use compare::score;pub use compare::verdict;pub use compare::verdict_for;pub use driver::DESCRIPTOR;pub use driver::METADATA;pub use nemotron::NEMOTRON_NANO_DESCRIPTOR;pub use nemotron::NEMOTRON_SUPER_DESCRIPTOR;
Modules§
- compare
- 2026-09-26: Given what each order produced, decide whether every reply is a function of its own request. The driver does the requests; this module does no I/O, so the tests construct each failure without a server.
- driver
- 2026-09-26: The part that talks to a server: issue the draw once per order,
keep every reply, and hand the lot to
compare. - nemotron
- 2026-10-03: Per-model gate ids for the KAT equality check on the Nemotron-3 models: the same
driver (
driver::KatEquality), each id with its own default subject in its model’s BENCH.toml, because a gate id has one required subject per box class (gate::check::record_is_required_subject). Promotion candidates until their BENCH entries are measured. - report
- 2026-09-26: How an equality run is presented: functions of the
super::compareresults, with no I/O.