Arena#
Evaluation tool: human pairwise preference voting, for quality that no automatic metric captures.
SIL-Wheel Arena is a human-powered pairwise preference evaluation system for comparing model outputs side by side. Annotators are shown the outputs of two anonymised models conditioned on the same input and select which they prefer. The system maintains a Glicko-2 leaderboard with 95% confidence intervals computed in closed form from each model’s rating deviation.
Arenas are self-contained and can be extended with new models and data at any point, enabling continuous evaluation alongside model development.
Reach for it when the thing you are judging is subjective, such as which caption reads better or which generated video looks more plausible. There is no ground truth to score against, so a human has to decide.
Reach for something else when you do have ground truth and a metric that measures it; Leaderboards gives you a number without needing a person in the loop.
The Arena dashboard: models ranked by ELO with their win, loss and tie counts, how those ratings moved as votes accumulated, and the latest votes.#
How It Works#
Each match presents a pair of outputs from two randomly selected models for the same input. Model identities are hidden so that the preference is based solely on output quality. Annotators choose from five options – strongly prefer A, prefer A, tie, prefer B, strongly prefer B – or mark both outputs as bad, or skip the match entirely.
A pairwise comparison match: the same input is shown once, with Model A’s output on the left and Model B’s output on the right. The annotator selects their preference using the buttons below.#
After a preference is recorded, both models’ ratings are updated using the Glicko-2 rating system, starting from a rating of 1500. Each vote is treated as its own rating period, updating each model’s rating, rating deviation (RD), and volatility. The five-way preference maps to a continuous outcome score (1.0 / 0.75 / 0.5 / 0.25 / 0.0), so partial preferences have a proportionally smaller effect on ratings than decisive ones. A “both bad” vote reduces both models’ ratings.
Supported Tasks#
An arena is configured through a manifest.json file that defines the inputs shown to the
annotator and the outputs produced by each model. Inputs and outputs can be any
combination of videos, images, and text. Common configurations include:
Video captioning. The input is a video clip, the output is a natural-language description. The annotator judges which caption better describes what is shown.
Generative model quality. The input is a text prompt or a reference image; the output is a synthesised video or image. The annotator judges visual quality and prompt fidelity.
VLM understanding. The input is a video or image with a question; the output is a structured or free-text answer. The annotator judges correctness and completeness.
The manifest also carries optional per-arena annotation instructions shown to the annotator during each match.
Arena Lifecycle#
Visibility. A newly created arena is unpublished and visible only to its owners and site admins. Once the arena is ready, an owner can publish it, making it visible to all authenticated users.
Adding models and data. The arena manifest lives on S3 and is periodically synced. To add a new model or new data items, update the manifest and upload the corresponding assets to S3, then trigger a manifest refresh from the arena management view. Existing votes remain valid; new models and items are incorporated into the sampling pool immediately.
Rating history. The leaderboard includes a per-model rating progression chart that replays all recorded votes in chronological order, showing how ratings evolved as more matches were played.
Vote export. All votes can be exported as a CSV file for offline analysis.