Expand description
2026-09-26: MLPerf Agentic Inference: teacher-forced replay of recorded
trajectories with inline scoring. The prompt for each client turn is the
recorded history plus the current client message (dataset), so the
model’s output is scored but never fed forward. Scoring ports the upstream
inline scorer’s two rules, workflow intent-code match and coding
bash-executable multiset IoU (scoring).
The dataset is not in the repository and nothing downloads it:
provision::ensure fails when dataset.jsonl is absent, naming the path,
and when it does not match a set expected_sha256. The BENCH.toml entry
(gate = "mlperf-agentic-subset") is status = "unmeasured" with no
metric bounds, gate::coverage::NOT_REQUIRED lists the gate with its
reason, and the verdict is always info (report.rs).
Sampling is not a parameter: every request carries the fixed set below plus a seed. A temperature-0 run would give numbers that look comparable to the MLPerf thresholds and are not.
Owner: bench, mlperf_agentic. Invariants: a completed run’s verdict is info, and a run with no scorable turn fails instead of reporting 0.0.
Modules§
- dataset
- 2026-09-26: The MLPerf Agentic Inference trajectory file: flat JSONL, one row per message, each conversation’s rows contiguous and numbered 1..=N in file order.
- provision
- 2026-09-26: Locate and verify the trajectory file,
dataset.jsonlin the artifact store’smlperf-agenticdirectory. Nothing downloads it: an operator places it there. - scoring
- 2026-09-26: Rust port of the comparison rules of the MLPerf Agentic
Inference inline scorer,
AgenticInferenceInlineScorerin mlcommons/endpoints@7935df4,src/inference_endpoint/evaluation/scoring.py: