Module scoring 

2026-09-26: Rust port of the comparison rules of the MLPerf Agentic Inference inline scorer, AgenticInferenceInlineScorer in mlcommons/endpoints@7935df4, src/inference_endpoint/evaluation/scoring.py:

• Workflow (^sim_\d+$ conversation ids): 1.0 if the intent code extracted from the model’s text is in the ground truth’s intent_codes, else 0.0.
• Coding (every other id): multiset IoU over normalized bash executables from bash tool calls.

scoring_tests.rs checks every function against assets/mlperf-agentic/parity_fixtures.json, whose expected values gen_parity_fixtures.py produced by executing the upstream class, including one case per alias-table entry and per shell wrapper. Cases a port could plausibly get wrong are among them: "reintent: I042" still scores through the bare-token fallback, and executables missing from the alias table vanish from both sides of the IoU.

One known divergence: Python’s \s also matches \x1c-\x1f, and char::is_whitespace does not. It affects only control characters between intent: and the code.

Owner: bench, mlperf_agentic. Invariants: for a gt_turn that has_ground_truth accepts, score_turn is in 0.0..=1.0.

Enums

Domain

2026-09-26: Which scoring rule applies, decided by conversation id: ^sim_\d+$ is workflow, anything else is coding.

Functions

bash_actions

2026-09-26: Normalized bash executables from a turn’s tool calls (upstream _bash_actions): only function.name == "bash" calls count; arguments may be an object or a JSON string (malformed JSON skips that call); command is preferred over cmd with Python truthiness (an empty command falls through). Per command: quoted spans stripped, split on || / | / && / ; / newline, leading env assignments and shell wrappers dropped, executable basenamed, lowercased, version-suffix-stripped and mapped through the alias table.

domain_of

ground_truth_intents

2026-09-26: Ground-truth intent codes from an expected assistant turn: strings from intent_codes, upper-cased, sorted and deduplicated, empty and non-string entries dropped. An empty set means the turn has no workflow ground truth and is excluded from the denominator.

has_ground_truth

2026-09-26: Whether an expected assistant turn carries scorable ground truth. Turns without it are excluded from the denominator, unlike issued but unanswered turns, which stay at 0.

model_intent

2026-09-26: The model’s intent code: the explicit intent: I123 form, searched over reasoning_content then content (the explicit pass covers both fields before the fallback runs over either), then the last bare I123 token as fallback. The explicit form is case-insensitive and upper-cased; the bare fallback is case-sensitive, as the fixture "maybe i123" pins.

multiset_iou

2026-09-26: sum(gt ∩ model) / sum(gt ∪ model) over action counts. The caller must pass a non-empty gt, so the union is not zero.

score_turn

2026-09-26: One turn’s score. gt_turn must be scorable (see has_ground_truth); a model turn that produced nothing scores 0 and stays in the denominator.