1. Addressability
Temporal RoPE offsets exceed the training range, so cached memories become unreadable even when they are physically present. Past the trained window, attention queries see phases the model never learned to address.
WorldTrace: Addressable Memory for Video World Models
ICML 2026 F2S Workshop, Best Paper
TL;DR
TL;DR. WorldTrace keeps compressed memory addressable with fixed in-distribution slot positions, then uses canonical-key writers for two goals: WorldTrace-Field for smoother long rollouts and WorldTrace-Landmark for recalling previously visited scenes, all without retraining the generator.
Abstract
We study visual persistence in interactive video world models, which rely on a Key–Value (KV) cache as a growing visual memory to carry forward previously generated frames. We find that models can no longer reliably address stored content once rollouts extend beyond the training horizon, because temporal RoPE offsets then fall outside the range seen during training. Moreover, naively compressing the cache in the RoPE-rotated space corrupts memory by averaging together incompatible positional phases. We propose WorldTrace, a training-free memory framework that keeps compressed memory addressable by assigning each summary slot a distinct, in-distribution virtual position. Within this addressable cache we study two compression approaches: WorldTrace-Field compresses history for temporal coherence, while WorldTrace-Landmark stores verbatim scene traces at detected transitions for episodic recall. To measure recall under controlled revisits, we introduce LoopBench, a benchmark evaluating whether a compressed cache can reconstruct a previously visited scene after a long detour. WT-Field improves temporal consistency by +15.5%, and WT-Landmark improves episodic recall by +19.5% on LoopBench, extending visually persistent generation without retraining.
Motivation
Autoregressive video world models promise interactive worlds, but visual persistence collapses once generation exceeds the training horizon.
Can an autoregressive video world model reliably remember where it has been, at any generation length?
Why memory breaks beyond the training horizon
Two coupled bottlenecks arise once generation crosses the training context window:
2. Content fidelity
Naive key averaging in RoPE-rotated space mixes incompatible phases. The resulting phase cancellation destroys the signal that compressed summaries are supposed to carry.
Method: WorldTrace
A two-tier KV cache: a verbatim recent window plus $N_s$ summary slots, with positions assigned by slot rank alone (independent of horizon) so every summary stays in-distribution at any generation length. Two complementary writers fill the slots:
Slot-rank position assignment
Virtual position for summary slot $s$:
$$t^{v}_{s} \;=\; q - \bigl(L_{\mathrm{attn}} - 1 - s\bigr), \quad s = 0,\ldots,N_s - 1$$
$q$ is the current query position, $L_{\mathrm{attn}} = N_s + N_r$ is the local attention window in latent frames, and $s$ indexes summary slots from oldest. The offset $(L_{\mathrm{attn}}-1-s)$ is fixed by slot rank alone, so slot positions depend on rank, not rollout length.
Canonical WT-Field writer
$$K_{\mathrm{field}}^{f}\!\left(t^{v}\right) \;=\; R\!\left(\theta_f\, t^{v}\right)\, \frac{1}{M}\sum_{m=1}^{M} R\!\left(-\theta_f\, t_m\right) K_{t_m}^{f}$$
For each RoPE frequency pair $f$, every source key is unrotated to its canonical content $R(-\theta_f t_m)K_{t_m}^{f}$, averaged in that space, then re-encoded at the slot's virtual position $t^{v}$. This avoids phase cancellation and preserves the mean attention score the $M$ source keys would receive at that shared position. Values carry no RoPE and are simply averaged. The $M$ frames per slot follow from the grouping, so $M \approx N_{\text{past}}/N_s$ grows with rollout length. WT-Field targets temporal coherence under compression; not a recall mechanism.
Frozen WT-Landmark writer
$$K_{\mathrm{land}}^{f}\!\left(t^{v}_{s}\right) \;=\; R\!\left(\theta_f\, t^{v}_{s}\right)\, R\!\left(-\theta_f\, t_{\ell^{*}}\right) K_{t_{\ell^{*}}}^{f}$$
Same form as WT-Field with $M=1$ and no averaging, where $t_{\ell^{*}}$ is the original timestamp of the selected landmark. Scene-entry frames are detected by a spike above threshold $\tau$ in the cosine distance between canonical keys of consecutive frames. The canonical key is computed once at landmark time and frozen, so each slot shift applies one fresh rotation rather than another unrotate→rerotate cycle, avoiding bfloat16 drift. Landmarks inherit slot-rank positions and target episodic recall over long rollouts.
Results
WorldTrace-Landmark: episodic recall
LoopBench tests episodic recall by asking the model to return to previously visited scenes and scoring the regenerated view with Position-Aligned CLIP (PAC). Across topology, path length, camera orientation, and multi-revisit settings, WT-Landmark consistently improves over sliding-window recall: 0.825 vs. 0.627 PAC on the long ABA path, 0.864 vs. 0.723 on standard ABA, and 0.941 vs. 0.892 on ABABA. The hardest $360^\circ$ pan shows the smallest gain (0.577 vs. 0.559), making the limitation visible rather than hidden by the aggregate.
Topology
Vary the number of intermediate waypoints before returning to the starting scene.
Edge length
Increase the number of generated chunks per leg to stretch context distance.
Orientation
Stress recall under camera-orientation changes, including wide pans.
Multi-revisit
Revisit the same place multiple times to test repeated episodic recall.
Pan recall videos
Pan path
Pan 1: Initial frame
Pan 1: Sliding window (baseline)
Pan 1: Ours
Pan 2: Initial frame
Pan 2: Sliding window (baseline)
Pan 2: Ours
Pan 3: Initial frame
Pan 3: Sliding window (baseline)
Pan 3: Ours
ABA recall videos
ABA path
ABA 1: Initial frame
ABA 1: Sliding window (baseline)
ABA 1: Ours
ABA 3: Initial frame
ABA 3: Sliding window (baseline)
ABA 3: Ours
WorldTrace-Field: coherence rollouts
Holding the content operator fixed (canonical averaging) and varying only the position assignment, slot-rank positions lead Block-Rel by +5.9% TempSSIM at 8× horizon and +2.8% at 16×. At 24× (N=48), WT-Field improves +15.5% TempSSIM over sliding-window while also lowering Local Scene Drift, where every $N$-dependent position formula degrades non-monotonically.
Coherence rollout videos
Sliding window
Block-Rel
Centroid
WT-Field
Takeaways
Diagnosis
- Long-horizon failure is a position problem, not a content problem.
- Naive averaging in RoPE-rotated space causes phase cancellation.
- Compression-only summaries collapse to a sliding window once their slots fall outside the trained range.
Method
- Slot-rank virtual positions: every summary stays in-distribution at any horizon.
- WorldTrace-Field: canonical-key averaging for coherence.
- WorldTrace-Landmark: frozen verbatim traces for recall.
Impact
- +15.5% TempSSIM at 24× training horizon (WT-Field).
- Higher PAC across LoopBench: WT-Landmark improves over sliding window in topology, edge-length, orientation, and multi-revisit scenarios.
- Training-free, $O(1)$ summary cache: drop-in for AR video world models.
Citation
@article{wu2026addressable,
title={Addressable Memory for Video World Models},
author={Wu, Xindi and Elflein, Sven and Lucas, James and Russakovsky, Olga and Leal-Taix{\'e}, Laura and Paschalidou, Despoina and Lorraine, Jonathan and O{\v{s}}ep, Aljo{\v{s}}a},
journal={arXiv preprint arXiv:2608.07408},
year={2026}
}