Module bfcl 

2026-09-26: BFCL v4 single-turn, full and subset draws, whole or sharded.

Its ground truth and AST checker live in bfcl-eval, so the work is split:

• Python materialises the dataset and scores the responses. Both scripts are committed in assets/bfcl/, written into the artifact store’s artifacts/bfcl, and run from a venv provisioned during load().
• Rust owns the draw, the generation, the streaming and the presentation, so a run reports its n (and warns when it is not the pinned size) before generation starts, and is cancellable between samples.

Generation sends one sample at a time.

Owner: bench, BFCL benchmark. Invariants: none beyond the types.

Re-exports

pub use report::MLPERF_FLOOR_CHECKPOINTS;

pub use report::MLPERF_FLOOR_NORMALIZED;

pub use report::MLPERF_FLOOR_OVERALL;

pub use report::is_mlperf_submission_checkpoint;

Modules

aggregate

2026-09-26: Combining BFCL shard results into the score the gate judges.

dataset

2026-09-26: Reading the materialised BFCL table and applying the draw to it. The draw takes the first n rows of each subset in file order, so the row order within a subset is the sample selection.

draw

2026-09-26: The BFCL sample draw. normalized_single_turn_score depends on the category mix, so a score is comparable only with scores on the same draw.

provision

2026-09-26: Provisions the BFCL artifacts under the plugin’s artifact directory on load(): writes the committed scripts, then finds a Python, builds a venv, pip-installs the pinned requirements.txt, runs the scorer’s --selftest, and materializes dataset.jsonl with provision.py. Each step reports a status before it runs. Everything after the script writes is skipped when the Stamp over the three shipped assets matches and the dataset and venv interpreter exist.

report

2026-09-26: How a BFCL run is presented (table, summary tiles, gate metrics) and its run verdict, including the MLPerf-edge floors.

sensitive

2026-09-26: The BFCL samples whose answer depended on what ran before them. Measured 2026-09-08: the golden n=995 draw run whole and the same draw run as four shards disagreed on exactly these sample_ids. The mechanism is cross-request SSM snapshot reuse, described in crate::gate::group. The driver warns on each of these it runs and reports the count as the known_partition_sensitive metric; membership does not change how a sample is scored.

Structs

Bfcl

Enums

Variant

Constants

ECHOLP_METADATA

FULL_DESCRIPTOR

FULL_METADATA

INHERIT_SHARD

2026-09-26: The shard parameter value that means “leave the current slice alone”. The note on the spec’s default says why it is a word and not an empty string or 0/1.

MAX_NEW_TOKENS

2026-09-26: The default per-sample output budget. kat_equality, which replays BFCL samples, takes its default budget from this constant too.

MINI_METADATA

MINI_NEMOTRON_NANO_DESCRIPTOR

2026-10-03: The mini draw on NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4, a separate gate id so the model has its own required subject (gate::check::record_is_required_subject).

MINI_NEMOTRON_SUPER_DESCRIPTOR

2026-10-03: The same on NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4.

SUBSET_DESCRIPTOR

SUBSET_ECHOLP_DESCRIPTOR

SUBSET_METADATA

SUBSET_MINI_DESCRIPTOR