Autoregressive Video Gen
Preface: AR in LLMs
Modern language models turn generation into a sequence of AR predictions. Instead of solving an entire response at once, an AR LLM factorizes it from left to right:
The causal attention mask is more than a modeling choice. It creates a systems contract: a generated token never depends on future tokens, so completed prefix states remain valid. The server can cache their keys and values, emit output immediately, and advance the sequence without recomputing the whole prefix.
Each new token reads the prefix, while the already emitted prefix stays fixed.
Past attention states become persistent session memory rather than repeated computation.
Streaming, cancellation, batching, and memory management operate at token boundaries.
This AR interface is the hidden reason LLM serving scales as a long-running system rather than a collection of fixed-length batch jobs. AR video generation asks whether video models can expose the same kind of stable boundary—at the level of frames or latent blocks—while retaining the visual quality of diffusion.
1. Why AR video generation matters
Full diffusion is a great clip generator—and a poor streaming primitive
A standard video diffusion transformer receives a noisy latent clip and repeatedly updates the whole clip with bidirectional spatiotemporal attention. This is a powerful inductive bias for a fixed-duration sample: every frame may use evidence from both past and future, and the model can repair globally inconsistent motion over multiple denoising iterations.
But the same coupling creates three deployment problems. First, the model cannot emit the beginning of the video while the future is still unresolved. Second, extending the duration increases the token set processed at every denoising step. Third, a prompt change in the middle of generation has no natural boundary: the sampler was designed to solve one global clip, not to maintain a live session.
AR diffusion factorizes time into frames or blocks:
This factorization introduces a sequential dependency across blocks, but it also creates a reusable boundary. Once block k is clean, it can be streamed to the user and summarized into a KV cache while block k+1 is denoised. The active working set can be bounded with a rolling window, and the prompt can be updated between blocks.
The wall-clock gap: bidirectional diffusion vs causal streaming
For a concrete long-video comparison, the LongLive paper uses SkyReels-V2. SkyReels-V2 is a Diffusion-Forcing system rather than a vanilla full-clip sampler, but its bidirectional attention preserves the same serving bottleneck: completed history cannot be reused as a KV cache under AR decoding. On a single H100, the reported gap for producing a 60-second video is: