Module score

Expand description

2026-09-26: Scoring for the agentic webserver task, ported from the harness’s score_run.py and followed_directions.py. Two separate axes:

webserver_ok, the outcome: the scorer builds and runs the code the agent left behind and asks /ping for pong, whether or not the agent built or checked anything itself.
followed_directions, the process: whether the commands the agent ran and the tree it left evidence each step in REQUIRED_STEPS.

Owner: bench, agentic. Invariants: followed_directions returns one entry per REQUIRED_STEPS name, in that order.

Structs§

REQUIRED_STEPS: 2026-09-26: The prompt-mandated process steps; Directions::overall is their AND.

followed_directions: 2026-09-26: Whether the agent performed each step in REQUIRED_STEPS. The evidence is the shell commands it ran and the tree it left, the two sources followed_directions.py uses.
free_port: 2026-09-26: An OS-assigned port, released on return, for a caller that binds it soon after (the self-started gate server in the server crate’s bench_selfstart and bench_lease). The scorer holds its port through the build with reserve_port instead.
per_step_tallies: 2026-09-26: How many iterations evidenced each step, keyed step:<name>, in the first iteration’s step order; empty for no iterations. followed_directions is all-or-nothing per iteration, so without this the tier record cannot say which step failed.
webserver_test: 2026-09-26: Builds the project in release, runs it on a reserved port and checks that /ping answers pong within serve_timeout. Every failure is reported in WebserverResult::error, not as an Err.