A Scalable Agentic Video Benchmark for Spatiotemporal Capability
1miHoYo Honkai AI R&D Team 2College AI, Tsinghua University 3University of Chinese Academy of Sciences 4MMLab, The University of Hong Kong 5National University of Singapore
*Equal contribution · †Corresponding author · ‡Work done while interning at miHoYo
Correspondence: yimingli9702@mail.tsinghua.edu.cn, yinhe.zheng@mihoyo.com
Understanding and tracking a dynamic world are fundamental to visual intelligence. Existing video benchmarks provide limited insight into whether models can maintain a coherent understanding of an evolving world state. We introduce Video-Arena, a benchmark that turns competitive games into scalable, verifiable tests of how well models understand evolving world states from videos. It covers three task categories: perceiving critical events, interpreting state updates, and tracking the resulting states. Our pipeline can generate videos and aligned reference labels from game records at scale. We release a golden set of 180 tasks across 9 games and evaluate models under two protocols: native visual understanding and tool-assisted agentic analysis. The best-performing systems achieve F1 scores of only 13.2 and 51.0, respectively. Native models struggle to perceive events reliably, while tool-assisted systems improve event recovery but still struggle to interpret state updates and track the resulting states consistently over time.
One case from each of the nine environments. As the replay plays, the ground-truth tuple for every critical event appears on the timeline: what happened (T1), how it moved the state (T2), and what the state is now (T3). Long matches are shown as a condensed excerpt; the clock shows real game time.
Every model watches the same case. Each mark is a real prediction from the model's submitted answer, placed at the moment the replay reaches it and judged against the ground truth with the strict tuple matcher (±3 s). The leaderboard is the running strict F1 over the clip.
Game records are replayed into videos, and every reference label is derived from the record itself, by a parser for event games or by an engine evaluator for board and card games. One adapter per game is all it takes to add a new arena.
A prediction counts only if its time falls within ±3 s of a GT event and every field of the tuple (event, update, state) is correct. Greedy one-to-one matching.
Native video understanding: frames sampled at 1 FPS, capped at 256 frames, a single answer in one pass.
Tool-assisted analysis inside a coding-agent harness: up to 200 turns and 2 hours of free exploration of the full video.
The best visual-protocol F1 is 13.2 (mean 4.6), while a temporal ceiling at the same 256-frame budget reaches 90.2. The frames contain the answer; the models miss it.
With a harness, Claude-Opus-5.0 jumps from 7.2 to 51.0 and the mean rises to 20.9. The gain comes from organising evidence, not from seeing more images.
On board and card games the best agent perceives nearly every move (T1 ≈ 1.00) yet tracks the resulting state far less reliably (T3 ≈ 0.52–0.68).
DOTA 2 and StarCraft II are both perception- and reasoning-bound: even the best agent reaches only T1 0.52 / 0.36, and 14.6 F1 on SC2.
Video-Arena is a purely scientific research project. We carry out no commercial activity of any kind with it: the benchmark is not sold, licensed for a fee, or used in any product or service.
Please read the licence and copyright terms of the upstream sources before use:
The paper's ethics statement lists the licence review for all nine environments. See the paper for details.
