metrale_bench/benchmarks/video/
mod.rs

1// SPDX-License-Identifier: MIT OR Apache-2.0
2
3//! 2026-09-26: The `video-fidelity` benchmark: did a clip's frames reach the
4//! model, in order?
5//!
6//! The order legs send one color sequence forwards (`01_colors_fwd.mp4`) and
7//! reversed (`02_colors_rev.mp4`), same container, duration and prompt, and
8//! require the reply to reverse with the clip. A no-video control runs last;
9//! if it names the whole sequence the run is VACUOUS (`score::verdict`).
10//!
11//! The other legs, all in `driver.rs`:
12//! - Geometry: the 1 s, 2 s and 4 s clips must satisfy
13//!   `t4 - t2 == 2 * (t2 - t1)` in prompt tokens
14//!   (`geometry::check_proportional`). The absolute count depends on the
15//!   server's `--video-fps`, which the benchmark neither sends nor reads.
16//! - Backend parity: the MP4 and the GIF of the same colors must cost the same
17//!   prompt tokens. The server decodes GIF in-process and every other
18//!   container with ffmpeg.
19//! - Mixed media: an image and a video in one request.
20//! - Integrity: two videos in one request, video before image, two opposite
21//!   clips in flight, and a video in an earlier turn.
22//! - Concurrency: C = 1, 2, 4 copies of one request, every reply correct and
23//!   every level at the C=1 prompt-token count.
24//!
25//! A leg whose request fails because the server cannot decode the container
26//! (`request::is_decoder_unavailable`) is Skipped, not failed; ffmpeg decoding
27//! is off unless the serve passes `--video-allow-ffmpeg`. A run where every leg
28//! was skipped is INCONCLUSIVE, not PASS.
29//!
30//! Registered in `registry.rs`; model targets gate on it through
31//! `gate = "video-fidelity"` entries in their BENCH.toml.
32//!
33//! Owner: bench, video.
34//! Invariants: none beyond the types.
35
36pub mod concurrency;
37pub mod driver;
38pub mod geometry;
39pub mod provision;
40pub mod request;
41pub mod score;
42
43pub use driver::{DESCRIPTOR, METADATA};
metrale_bench/benchmarks/video/mod.rs

metrale_bench/benchmarks/video/
mod.rs
