1The University of Hong Kong 2LIGHTSPEED 3The Chinese University of Hong Kong 4Tsinghua University
‡ Project Leader † Corresponding Author
Vision-language model (VLM) agents are increasingly deployed in interactive game environments. Yet game benchmarks for VLM agents typically report a single first-attempt score per (agent, game) pair, focus on single-agent Solo play, and lack unified protocols for evaluating heterogeneous agent classes (commercial VLMs, open-weight VLMs, and specialized game policies) on the same footing.
We address these gaps with OmniGameArena, a real-time benchmark of twelve newly built Unreal Engine 5 games spanning Solo (7), PvP (3), and Coop (2) with unified action interfaces, and the Improvement Dynamics Curve (IDC), an agentic-reflection harness in which a tool-using reflector LLM autonomously refines a bounded skill prompt across multiple rounds. Beyond cold-start leaderboard scores, IDC exposes two additional observables for each (agent, game) pair: how the score evolves across reflection rounds, and how the learned skill behaves on held-out task variants. We report these observables for twelve VLM agents on the cold-start leaderboard and four top agents under IDC.
Solo, PvP, and Coop regimes share a unified action interface. Every game instance (rules, layouts, scripts, scoring) is authored from scratch for this benchmark rather than reused from public titles, which lowers pre-training leakage and contamination.
An agentic-reflection framework whose autonomous tool-use reflector refines a bounded skill prompt across rounds, with persistent memory and best-skill rollback, turning each instance into an improvement trajectory.
Across games and agents, leadership rotates and no single VLM dominates; origin-task improvement does not by itself predict held-out variant transfer, a divergence that cold-start leaderboard scores hide but multi-round reflection brings out.
Each game targets a different mix of visual perception, spatial navigation, reaction, memory, planning, adversarial interaction, and cooperation.
Each (agent, game) instance runs for multiple rounds. The agent plays K episodes under a current skill prompt; a tool-using reflector then inspects the trajectories, deciding on its own what to read and when to stop, before refining the skill for the next round.
Inspect, search, and review the K episode trajectories from the current round.
Compare causes across episodes and explain the success and failure cases.
Test, verify, and confirm candidate skills before they are committed to the persistent skill set.
Summarize, refine, and store the refined skill prompt for the next round.
Leaderboard
Mean normalized score · Solo · Quality (PDQ)
Head-to-head win-rates across all PvP pairings: each cell is the row model (Player 1) vs. the column model (Player 2).
Gameplay recordings
Browse cold-start episodes by game. Solo and Coop show every model; PvP is a head-to-head matchup viewer. Numbers are normalized scores.
A tool-using reflector distills a skill prompt from the agent's own episodes.
Below: the learned skill, how its score evolves across reflection rounds, and how the skill transfers to three held-out task variants (with vs. without skill).
Leadership rotates across games; commercial agents hold a wide gap over open-weight VLMs and specialized policies.
All four top agents improve over their cold-start baseline through reflection, yet peak performance is typically reached before the final round.
Origin-task improvement and held-out variant transfer can diverge, a divergence that cold-start leaderboard scores do not reveal.
@article{lin2026omnigamearena,
title = {OmniGameArena: A Unified UE5 Benchmark for VLM Game Agents with Improvement Dynamics},
author = {Lin, Mingxian and Qian, Shengju and Liu, Yuqi and Huang, Yi-Hua and Wang, Yiyu and Huang, Wei and Li, Yitang and Zhang, Fan and Hu, Zeyu and Zhu, Lingting and Wang, Xin and Qi, Xiaojuan},
journal = {arXiv preprint arXiv:2606.09826},
year = {2026}
}