Module agentic 

2026-09-26: The agentic-webserver benchmark: N iterations of one task (write a Rust Axum ping/pong project, test it, run it, curl it, tear it down), each in a fresh sandbox. After each iteration the scorer builds and runs what the agent left and asks /ping for pong, and checks the agent’s commands against the prompt’s directions.

The prompt is verbatim from bench/fp8_dgx2_drift/harness/run_tier.sh; the agent loop is our own, not the opencode client that script drives. It runs model-authored shell; see agent for the containment.

Owner: bench, agentic. Invariants: none beyond the types.

Modules

agent

2026-09-26: The agentic benchmark’s agent loop: tool-calling turns against the served endpoint, with the tools run inside a per-iteration sandbox directory. Its tools, prompt and caps are modelled on the opencode client that bench/fp8_dgx2_drift/harness/run_tier.sh drives.

preflight

2026-09-26: Warm-up sanity check before a tier: the endpoint must answer 4 to 2+2, or the run stops. A port of warmup_endpoint in bench/fp8_dgx2_drift/harness/run_tier.sh. A 200 from /v1/models proves only that a server is listening; this proves the checkpoint still decodes.

score

2026-09-26: Scoring for the agentic webserver task, ported from the harness’s score_run.py and followed_directions.py. Two separate axes:

warm

2026-09-26: The shared warm cargo target dir, a port of the harness’s warm_cargo_cache.sh. With a cold dir every generated Axum project compiles its whole dependency tree inside the scorer’s build timeout. This builds a template project in the debug profile (for the agent’s own cargo test and cargo run) and the release profile (for the scorer’s cargo build --release), and returns the dir that the agent shell and the scorer both set as CARGO_TARGET_DIR.

Structs

AgenticWebserver

Constants

DESCRIPTOR

2026-09-26: Under --pull-request-gate each model’s serve recipe and metric bounds come from its BENCH.toml entry for this gate; for the 35B MoE that is recipe qwen3.6/qwen3.6-35b-a3b-fp8-bf16head.

METADATA

PROMPT

2026-09-26: The task, verbatim from run_tier.sh’s PROMPT. Changing a word changes the benchmark.