Module aggregate 

2026-09-26: Combining BFCL shard results into the score the gate judges.

Owner: bench, BFCL benchmark. Invariants: aggregate depends only on each subset’s summed (hits, n), so every partition of the same rows gives the same result through union.

Why shard scores cannot simply be averaged

score.py does not compute a flat mean. It aggregates hierarchically, and the weights are the whole difficulty:

non_live      = mean[ mean(simple_python, simple_java, simple_javascript),
                      multiple, parallel, parallel_multiple ]      <- four terms
live          = sample-weighted mean over its subsets
hallucination = mean(irrelevance, live_irrelevance)                <- unweighted
normalized    = mean(the categories present)
overall       = flat mean over every scored sample

So simple_javascript (31 rows in the golden draw) weighs the same as simple_python (248 rows), a third of one term each, and irrelevance (24 rows) counts equally with live_irrelevance (88). A mean of the shards’ normalized_single_turn_score is therefore not the whole-set value, and overall_accuracy survives only a sample-count-weighted mean, to the 2 decimal places each shard’s JSON is rounded to.

score.py also builds each category from the subsets present, so a subset missing from a shard changes that category’s divisor. live_parallel is 16 rows in the golden draw; a quarter of it is four.

What this does instead

Shards report per-subset (hits, n) integer counts (subset_totals); every sample scores exactly 0.0 or 1.0 in score.py, so nothing is lost. Summing them across shards and aggregating once applies the weighting to the union, never to a partial view.

The arithmetic below is a second implementation of score.py’s. A test in provision.rs runs the real score.py on a 12-sample fixture and checks aggregate against its output.

Structs

Aggregate

2026-09-26: The numbers the gate judges.

Tally

2026-09-26: Hits and sample count for one subset.

Constants

HALLUCINATION

2026-09-26: score.py’s hallucination subsets.

LIVE

2026-09-26: score.py’s live subsets.

NON_LIVE

2026-09-26: score.py’s non_live subsets, in its order.

SIMPLE_AST

2026-09-26: The three simple_* subsets collapse to one term inside non_live.

Functions

aggregate

2026-09-26: Aggregates per-subset tallies with score.py’s weighting. Given the union of every shard’s tallies it scores the whole draw; given one run’s tallies it scores that run, which is how the provision.rs test compares it with score.py.

tallies_from_metrics

2026-09-26: Recovers per-subset tallies from a gate record’s metrics, the subset.<name>.hits and subset.<name>.n keys that report.rs writes. None when the record has no tallies, or a subset has hits but no n. A caller treats None as “cannot be aggregated”, never as an empty contribution, which would score the group over fewer samples than the draw.

union

2026-09-26: Sums shard tallies per subset. A missing or duplicated shard is not detected here.