Expand description
2026-09-26: Cross-request and cross-path integrity checks, used by the vision and video benchmarks. They target machinery a single well-formed request cannot reach, whose failure is a fluent answer to the wrong input rather than an error:
heterogeneous_concurrency: several different requests in flight at once, each required to get its own answer.cache_leak: the same prompt text with a different image, back to back.long_prompt_path: the same image and question with filler prepended.media_in_history: media in an earlier turn or a tool result.stream_parity: the same request streamed and blocking.multi_choice:n > 1with an image.responses_parityandthinking_parity(theresponsessubmodule).
Owner: bench, media integrity.
Invariants: a check whose request failed never returns Cell::Pass.
Enums§
- Cell
- 2026-09-26: Outcome of one integrity check.
Functions§
- cache_
leak - 2026-09-26: The same prompt text with a different image, back to back.
- heterogeneous_
concurrency - 2026-09-26: Several different requests at once, each scored against its own expected answer.
- image_
request - 2026-09-26: One image part plus a prompt, as a streaming chat body with thinking off.
- long_
prompt_ path - 2026-09-26: The same image and question, short and with a large block of
filler text prepended; both replies must satisfy
want. - media_
in_ history - 2026-09-26: Media that is not in the message asking the question: in an
earlier turn or in a tool result. The server collects media from every
message, tool results included (
api/chat/msg_entry.rs,collect_message_media); a regression that scanned only the final message would pass the legs that put the image and the question in one turn. - multi_
choice - 2026-09-26:
n > 1with an image, on the blocking path. - responses_
image_ request - 2026-09-26: A blocking Responses-API request carrying one image.
- responses_
parity - 2026-09-26: The Responses API sees the image, and sizes it the same.
- stream_
parity - 2026-09-26: The same request streamed and not streamed.
- thinking_
image_ request - 2026-09-26: The same image request with thinking on.
- thinking_
parity - 2026-09-26: Vision with thinking on, through chat completions.
Type Aliases§
- Subject
- 2026-09-26: One concurrent subject: the request, a predicate scoring its own reply, and a label for the failure message.