TL;DR.WorldTrace keeps a video world model's compressed memory addressable, by giving every summary slot an in-distribution virtual position, and informative, by merging keys only after their rotation is removed. The model's own attention decides which observations to keep verbatim and which to merge. On LoopBench it improves average DINOv2 recall by 32.6% over a sliding window at the same attended-cache size, without retraining the generator.
Abstract
We study visual persistence in interactive video world models. These models rely on a Key-Value (KV) cache as a growing visual memory to carry forward previously generated frames. However, we find that models can no longer reliably address stored content once rollouts extend beyond the training horizon, because temporal Rotary Positional Embedding (RoPE) offsets can exceed the training range and the model struggles to retrieve the relevant visual information through attention. Moreover, naively compressing the cache in the RoPE-rotated space degrades memory by averaging keys with incompatible positional phases. To address this, we propose WorldTrace, a training-free memory framework for long-horizon visual persistence. WorldTrace keeps compressed memory addressable by assigning each summary slot a distinct, in-distribution virtual position. Within this cache, WorldTrace uses the model's own attention to decide which observations to retain verbatim and which redundant ones to merge into compressed summaries, all within one bounded addressable memory bank. We evaluate this ability with LoopBench, a benchmark that tests whether a compressed cache can reconstruct a previously visited scene after a long detour. On LoopBench, WorldTrace improves average DINOv2 recall similarity by 32.6% over a sliding window at the same cache budget, extending visually persistent generation without retraining.
Motivation
Autoregressive video world models promise interactive worlds, but visual persistence degrades rapidly once generation exceeds the training horizon.
Can an autoregressive video world model reliably remember where it has been over long horizons?
Why memory breaks beyond the training horizon
Two coupled bottlenecks arise once generation crosses the training context window:
1. Addressability
Temporal RoPE offsets can exceed the training range, so attention-based retrieval of cached memories becomes unreliable even when they are still stored. Past the trained window, some temporal RoPE components take on rotation angles the model never saw during training.
2. Informativeness
Averaging keys in RoPE-rotated space sums vectors rotated by different angles. Keys from distant timestamps can partially cancel, weakening each frequency's contribution and distorting the compressed summary.
The two failures are coupled: position assignment and compression act on the same query–key scores. WorldTrace combines in-range virtual positions with averaging in a common (canonical) coordinate system, so retaining distant content does not require retaining its original temporal offset.
Method: WorldTrace
A two-tier KV cache: a verbatim recent window plus $N_s$ summary slots, with positions assigned by slot rank, at offsets from the query that do not change with horizon, so every summary stays in-distribution at any generation length. Memory compression becomes a structured projection of the growing visual history into a fixed set of retrievable traces, a form of structured sparse attention: redundant content is averaged into compact summaries, distinctive observations are kept verbatim, and both share one position assignment that keeps them addressable.
WorldTrace overview. WorldTrace partitions the context window into a compressed summary cache $S$ for the distant past and a verbatim recent window $R$ (top). Each slot occupies one latent-frame position in the cache, with recent slots holding one verbatim frame and summary slots holding compressed history. In WorldTrace-Field (bottom left), keys are unrotated and averaged in canonical space, then re-rotated at each slot's virtual position. In WorldTrace-Landmark (bottom right), scene-entry landmarks store frozen canonical keys and are re-rotated to the current virtual position.
$q$ is the current query position, $L_{\mathrm{attn}} = N_s + N_r$ is the local attention window in latent frames, and $s$ indexes summary slots from oldest. The offset $(L_{\mathrm{attn}}-1-s)$ is fixed by slot rank alone: advancing $q$ shifts every slot's virtual position by the same amount, so each slot's offset from the query depends on its rank, not on rollout length. With $L_{\mathrm{attn}}-1 \le \Delta t_{\mathrm{train}}$ (5 for Matrix-Game-2), the oldest slot sits at offset $L_{\mathrm{attn}}-1$ and the newest at $N_r$, both inside the trained range.
For each temporal RoPE frequency pair $f$, every source key is unrotated to its canonical content $R(-\theta_f t_m)K_{t_m}^{f}$, averaged in that space, then re-encoded at the slot's virtual position $t^{v}$. This avoids phase cancellation and preserves the mean query–key score the $M$ source keys would receive if all were placed at that shared virtual position. Values carry no RoPE and are simply averaged. The $T-N_r$ frames outside the recent window are split into $N_s$ contiguous groups, so each slot averages $M \approx (T-N_r)/N_s$ frames, where $T$ is the number of latent frames generated so far; $M$ grows with rollout length. On its own, WorldTrace-Field stays near the sliding window on LoopBench (0.576 vs. 0.587 tier-averaged DINOv2); in the full method its summaries hold redundant history while retained frames carry distinctive scenes.
Same form as WorldTrace-Field with $M=1$ and no averaging, where $t_{\ell^{*}}$ is the original timestamp of the selected landmark. A frame whose canonical keys have cosine distance above threshold $\tau$ from the preceding frame's marks a scene entry; the $N_s$ slots hold the most recent scene-entry frames (repeating the oldest until $N_s$ have been seen), and the oldest landmark is evicted when a new one arrives. The canonical key is computed once at landmark time and frozen, so every read applies a single fresh rotation to the current virtual position instead of repeated unrotate→re-rotate cycles, which accumulate floating-point error in bfloat16. Landmarks take slot-rank positions and target episodic recall over long rollouts.
Combined WorldTrace
Individual observations and compressed summaries share one bounded memory bank, read through the same attention. Candidate merges and evictions are scored by the attention-response error they cause on sampled queries, and key–value similarity decides how the selected entries are combined: compatible observations are averaged into a count-weighted field summary, and an observation kept without merging is a landmark. The evaluated implementation keeps a bank of verbatim frames alongside the summaries, and the model's attention queries select which entry to expose.
Results
LoopBench: returning to a previously visited scene
LoopBench sends the model along a path through waypoints and back to a place it generated earlier, then compares the return view with that earlier view: DINOv2 scene consistency (↑) for scene features and Return LPIPS (↓) for perceptual appearance. It covers four task families with three conditions each. On Matrix-Game-2 (1.3B), with the same six-frame attended window for every method, WorldTrace has the highest mean DINOv2 score of the compared methods in all 12 tier/condition entries and raises the tier-averaged DINOv2 score from 0.587 to 0.778 over the sliding-window cache: +32.6% recall, without retraining.
Topology
Vary the number of intermediate waypoints before returning to the starting scene.
Rollout length
Stretch the ABA loop from 8 to 16 to 32 generated chunks.
Orientation
The agent stays at A while the camera pans 90° or 180° and back, or makes one full 360° turn.
Multi-revisit
Paths with more than one revisit: ABABA returns to A twice; ABCBA and ABCDBA pass through B again before the final return to A. Scores compare the frames generated at A on the first visit and on the final return.
LoopBench benchmark gallery. Solid and dashed blue arrows mark the outbound and return legs, lighter arcs mark a second traversal, and dotted arrows mark shortcuts that skip intermediate waypoints. Row 1 shows topology (ABA, ABCA, ABCDA). Row 2 varies rollout length on the ABA topology, increasing the interval between departure and final return. Row 3 varies camera orientation at A. The 90° and 180° pans reverse to the initial orientation, whereas the 360° condition completes one revolution. Row 4 shows multi-revisit patterns, where $R$ counts returns to A in ABABA, and visits to B in the ABCBA palindrome and the ABCDBA shortcut. These pattern-specific counts are all two.
DINOv2 scene consistency (↑) / Return LPIPS (↓) between the frames generated at the revisited location (e.g., A in ABA) on the first visit and on the return, on Matrix-Game-2, mean over 51 rollouts (17 scenes × 3 seeds). $N$ is rollout length in AR chunks. Every method uses the same six-frame attended window ($N_s=4$, $N_r=2$ for WorldTrace, whose row is the combined variant). Total memory use differs: WorldTrace's bank of four verbatim frames adds a constant 3.1 GB of GPU memory, and scoring it adds 27% wall-clock time (paper appendix). Best value per condition in bold.
Topology
ABA N=16
ABCA N=17
ABCDA N=16
Sliding window
0.577 / 0.643
0.627 / 0.643
0.599 / 0.640
MemRoPE
0.623 / 0.621
0.630 / 0.630
0.607 / 0.626
WorldTrace
0.798 / 0.512
0.767 / 0.548
0.805 / 0.489
Rollout length
ABA short N=8
ABA N=16
ABA long N=32
Sliding window
0.805 / 0.525
0.577 / 0.643
0.422 / 0.700
MemRoPE
0.816 / 0.505
0.623 / 0.621
0.464 / 0.694
WorldTrace
0.891 / 0.460
0.798 / 0.512
0.750 / 0.537
Camera orientation
Pan 90° N=4
Pan 180° N=8
Pan 360° N=8
Sliding window
0.838 / 0.561
0.560 / 0.634
0.406 / 0.740
MemRoPE
0.844 / 0.547
0.561 / 0.639
0.394 / 0.742
WorldTrace
0.911 / 0.453
0.784 / 0.649
0.423 / 0.722
Multi-revisit
ABABA N=32
ABCBA N=20
ABCDBA N=20
Sliding window
0.462 / 0.692
0.571 / 0.659
0.595 / 0.653
MemRoPE
0.496 / 0.675
0.584 / 0.642
0.602 / 0.634
WorldTrace
0.791 / 0.518
0.807 / 0.515
0.808 / 0.503
LPIPS is lower than the sliding window in every condition except Pan 180° (0.649 vs. 0.634), and the full 360° pan remains the hardest case (0.423 vs. 0.406 DINOv2, a gap within one standard error). MemRoPE, a training-free positional baseline, stays within 0.046 DINOv2 of the sliding window.
Sliding window
0.587
default cache, evicts the oldest frames
WorldTrace-Field
0.576
compressed summaries only
WorldTrace-Landmark
0.715
one verbatim frame only
WorldTrace
0.778
both, selected by attention
Averaged summaries alone stay near the sliding window, a single verbatim landmark recovers most of the gap, and combining both is best on average (tier-averaged DINOv2 computed from the paper's Tables 1 and 2; differences within single conditions can be small). The advantage persists at long horizons, though not monotonically: at 128 chunks, WorldTrace reaches 0.698 DINOv2 / 0.572 LPIPS, against 0.251 / 0.751 for the sliding window.
Pan 90° recall videos
Clips compare the sliding window with WorldTrace-Landmark, as in the paper's appendix figures.
Pan path
Pan 1: Initial frame
Pan 1: Sliding window (baseline)
Pan 1: WorldTrace-Landmark (ours)
Pan 2: Initial frame
Pan 2: Sliding window (baseline)
Pan 2: WorldTrace-Landmark (ours)
Pan 3: Initial frame
Pan 3: Sliding window (baseline)
Pan 3: WorldTrace-Landmark (ours)
ABA recall videos
Clips compare the sliding window with WorldTrace-Landmark, as in the paper's appendix figures.
ABA path
ABA 1: Initial frame
ABA 1: Sliding window (baseline)
ABA 1: WorldTrace-Landmark (ours)
ABA 2: Initial frame
ABA 2: Sliding window (baseline)
ABA 2: WorldTrace-Landmark (ours)
Takeaways
Diagnosis
Memory must stay addressable: beyond the training horizon, temporal RoPE offsets can leave the trained range and attention-based retrieval of stored frames becomes unreliable.
Memory must stay informative: averaging keys in RoPE-rotated space can partially cancel keys from distant timestamps.
The two failures are coupled: position assignment and compression act on the same query–key scores.
Method
Slot-rank virtual positions: every summary stays in-distribution at any horizon.
Canonical-key averaging for redundant history (WorldTrace-Field) and frozen verbatim landmarks for distinctive scenes (WorldTrace-Landmark).
The model's attention selects which retained frame to expose; no retraining.
Impact
+32.6% DINOv2 recall over a sliding window on LoopBench, at the same attended-cache size.
Highest mean DINOv2 of the compared methods (sliding window, MemRoPE) in all 12 LoopBench tier/condition entries on Matrix-Game-2.
At 128 chunks: 0.698 vs. 0.251 DINOv2 against the sliding window.
Citation
@inproceedings{wu2026addressable,
title={Addressable Memory for Video World Models},
author={Wu, Xindi and Elflein, Sven and Lucas, James and Russakovsky, Olga and Leal-Taix{\'e}, Laura and Paschalidou, Despoina and Lorraine, Jonathan and O{\v{s}}ep, Aljo{\v{s}}a},
booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
year={2026},
eprint={2608.07408},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2608.07408}
}
We thank our colleagues at NVIDIA's Spatial Intelligence Lab and at Princeton University for many helpful discussions and valuable feedback. This work was supported by NVIDIA.