.. SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. .. SPDX-License-Identifier: Apache-2.0 .. .. Licensed under the Apache License, Version 2.0 (the "License"); .. you may not use this file except in compliance with the License. .. You may obtain a copy of the License at .. .. http://www.apache.org/licenses/LICENSE-2.0 .. .. Unless required by applicable law or agreed to in writing, software .. distributed under the License is distributed on an "AS IS" BASIS, .. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. .. See the License for the specific language governing permissions and .. limitations under the License. Arena ===== **Evaluation tool: human pairwise preference voting, for quality that no automatic metric captures.** SIL-Wheel Arena is a human-powered pairwise preference evaluation system for comparing model outputs side by side. Annotators are shown the outputs of two anonymised models conditioned on the same input and select which they prefer. The system maintains a Glicko-2 leaderboard with 95% confidence intervals computed in closed form from each model's rating deviation. Arenas are self-contained and can be extended with new models and data at any point, enabling continuous evaluation alongside model development. **Reach for it when** the thing you are judging is subjective, such as which caption reads better or which generated video looks more plausible. There is no ground truth to score against, so a human has to decide. **Reach for something else when** you do have ground truth and a metric that measures it; :doc:`leaderboard` gives you a number without needing a person in the loop. .. figure:: /_static/images/arena.png :alt: The Arena dashboard with an ELO ranking table, a rating-over-votes chart, and a recent votes log The Arena dashboard: models ranked by ELO with their win, loss and tie counts, how those ratings moved as votes accumulated, and the latest votes. How It Works ------------ Each match presents a pair of outputs from two randomly selected models for the same input. Model identities are hidden so that the preference is based solely on output quality. Annotators choose from five options -- strongly prefer A, prefer A, tie, prefer B, strongly prefer B -- or mark both outputs as bad, or skip the match entirely. .. figure:: /_static/images/arena_comparison.png :alt: An Arena comparison with the driving clip and question at the top and two anonymised model responses side by side under Model A and Model B A pairwise comparison match: the same input is shown once, with Model A's output on the left and Model B's output on the right. The annotator selects their preference using the buttons below. After a preference is recorded, both models' ratings are updated using the Glicko-2 rating system, starting from a rating of 1500. Each vote is treated as its own rating period, updating each model's rating, rating deviation (RD), and volatility. The five-way preference maps to a continuous outcome score (1.0 / 0.75 / 0.5 / 0.25 / 0.0), so partial preferences have a proportionally smaller effect on ratings than decisive ones. A "both bad" vote reduces both models' ratings. Supported Tasks --------------- An arena is configured through a ``manifest.json`` file that defines the inputs shown to the annotator and the outputs produced by each model. Inputs and outputs can be any combination of videos, images, and text. Common configurations include: - **Video captioning.** The input is a video clip, the output is a natural-language description. The annotator judges which caption better describes what is shown. - **Generative model quality.** The input is a text prompt or a reference image; the output is a synthesised video or image. The annotator judges visual quality and prompt fidelity. - **VLM understanding.** The input is a video or image with a question; the output is a structured or free-text answer. The annotator judges correctness and completeness. The manifest also carries optional per-arena annotation instructions shown to the annotator during each match. Arena Lifecycle --------------- **Visibility.** A newly created arena is unpublished and visible only to its owners and site admins. Once the arena is ready, an owner can publish it, making it visible to all authenticated users. **Adding models and data.** The arena manifest lives on S3 and is periodically synced. To add a new model or new data items, update the manifest and upload the corresponding assets to S3, then trigger a manifest refresh from the arena management view. Existing votes remain valid; new models and items are incorporated into the sampling pool immediately. **Rating history.** The leaderboard includes a per-model rating progression chart that replays all recorded votes in chronological order, showing how ratings evolved as more matches were played. **Vote export.** All votes can be exported as a CSV file for offline analysis.