Autoregressive Video Gen

Preface: AR in LLMs

Modern language models turn generation into a sequence of AR predictions. Instead of solving an entire response at once, an AR LLM factorizes it from left to right:

The causal attention mask is more than a modeling choice. It creates a systems contract: a generated token never depends on future tokens, so completed prefix states remain valid. The server can cache their keys and values, emit output immediately, and advance the sequence without recomputing the whole prefix.

This AR interface is the hidden reason LLM serving scales as a long-running system rather than a collection of fixed-length batch jobs. AR video generation asks whether video models can expose the same kind of stable boundary—at the level of frames or latent blocks—while retaining the visual quality of diffusion.

1. Why AR video generation matters

Full diffusion is a great clip generator—and a poor streaming primitive

A standard video diffusion transformer receives a noisy latent clip and repeatedly updates the whole clip with bidirectional spatiotemporal attention. This is a powerful inductive bias for a fixed-duration sample: every frame may use evidence from both past and future, and the model can repair globally inconsistent motion over multiple denoising iterations.

But the same coupling creates three deployment problems. First, the model cannot emit the beginning of the video while the future is still unresolved. Second, extending the duration increases the token set processed at every denoising step. Third, a prompt change in the middle of generation has no natural boundary: the sampler was designed to solve one global clip, not to maintain a live session.

AR diffusion factorizes time into frames or blocks:

This factorization introduces a sequential dependency across blocks, but it also creates a reusable boundary. Once block k is clean, it can be streamed to the user and summarized into a KV cache while block k+1 is denoised. The active working set can be bounded with a rolling window, and the prompt can be updated between blocks.

The wall-clock gap: bidirectional diffusion vs causal streaming

For a concrete long-video comparison, the LongLive paper uses SkyReels-V2. SkyReels-V2 is a Diffusion-Forcing system rather than a vanilla full-clip sampler, but its bidirectional attention preserves the same serving bottleneck: completed history cannot be reused as a KV cache under AR decoding. On a single H100, the reported gap for producing a 60-second video is:

The gap cannot be attributed to distillation alone. Once past blocks become reusable state, LongLive no longer revisits the entire video prefix at every update. The comparison is not a controlled AR-versus-bidirectional ablation—resolution, kernels, and end-to-end accounting differ—but it makes the systems point visible: few-step distillation shrinks the inner solver; AR execution removes repeated work around that solver.

The ≈46-second LongLive value is derived from the paper's native 16-FPS output and 20.7 generated FPS: 60 × 16 ÷ 20.7 ≈ 46 seconds. See the LongLive comparison page.

The catch is equally important: AR generation conditions on its own past. A small error becomes context for the next block, and causal attention removes the future information that helped a bidirectional model repair a clip. It is therefore useful to separate two questions:

• AR changes the temporal interface from “solve a clip” to “advance a stream.”
• Per-block compute determines whether advancing that stream is actually fast enough for interaction.

AR does not imply few-step generation, and distillation is not the definition of an AR model. Distillation is one practical route for converting a strong bidirectional model into an AR student while reducing the number of DiT evaluations; Section 4 returns to that distinction.

2. AR video generation reuses LLM inference infrastructure

Once video generation is expressed as a persistent AR decode loop, much of the LLM serving playbook becomes relevant. The transferable idea is not “video is just text.” It is that both workloads expose an immutable prefix, reusable per-session state, an incremental output boundary, and a scheduler that repeatedly advances active requests.

How AR video makes KV reuse possible

At every self-attention layer, a completed video block produces keys and values that summarize how later blocks may attend to it. With a causal attention mask, block k cannot change the hidden states of blocks < k. Their K/V tensors therefore become immutable session state: every denoising evaluation for the next block reads the same history instead of projecting the entire prefix again. A bidirectional diffusion model does not expose this boundary—the representation of an earlier frame can change when future noisy tokens change—so its old K/V is generally stale after the next global update.

The factor of two stores both keys and values. Video makes this state unusually expensive because every temporal block contains many spatial latent tokens: increasing duration grows N_cached by far more than one token per frame. KV reuse removes projection compute, but a practical system also needs an explicit policy for when cached state is committed, quantized, evicted, or rebuilt.

Long video needs a cache policy, not an ever-growing cache

Keeping every historical token makes KV memory grow linearly with duration. A sink-plus-window layout changes the live state from the full history O(T) to a fixed set of S sink blocks plus W recent blocks, or O(S + W). The recent window preserves motion continuity; the sink carries a compact long-range anchor such as identity, composition, or the beginning of a shot. The price is lossy history: a window that is too short saves memory and attention work but can forget details that no sink represents.

LongLive 1.0 demonstrates this trade-off with short-window attention plus a frame sink. LongLive-2.0 makes the policy shot-aware: a global sink persists as an identity anchor, while a shot-level sink is rebound at scene changes. It also stores the KV cache in NVFP4 and reconstructs cached chunks with parallel dequantization, treating cache bandwidth and capacity as first-class inference bottlenecks.

3. Early attempts in AR training

A video model can become AR in several ways: train an AR architecture from scratch, fine-tune a pretrained model with a causal attention mask and AR objective, or distill a bidirectional teacher into an AR student. Early work favored distillation because the best visual priors already lived inside expensive bidirectional diffusion models.

In this setting, distillation performs two jobs at once: it transfers quality across a change in information flow, and it compresses a many-step diffusion solver into a few-step generator. That makes distillation a practical route to AR generation—not the only definition or implementation of AR training.

CausVid: convert first, then distill

CausVid established a two-stage recipe. It first initializes an AR student from probability-flow ODE trajectories produced by a pretrained teacher. It then uses an asymmetric form of distribution matching distillation (DMD): the generator uses causal attention, while the stronger critic-like teacher remains bidirectional. The result compresses a 50-step teacher into a four-step AR generator and uses KV caching for streaming inference.

DMD is attractive because it asks the student distribution—not every individual sample—to match a target distribution. In practice, the training loop maintains a generator, a real-score model tied to the teacher distribution, and a fake-score model that tracks the student's current distribution. Their score difference pushes samples toward the real distribution.

Self Forcing: train where the model will actually live

AR generation creates exposure bias. Teacher-forced training shows the model clean ground-truth history; inference shows it imperfect self-generated history. Self Forcing makes this mismatch the center of the algorithm: it rolls the student forward with its own KV cache, evaluates those self-generated sequences with a video-level DMD objective, and uses stochastic gradient truncation to keep training tractable.

This is the first major conceptual shift in the story: few-step quality is not enough; an AR model must be good on the states created by its own previous mistakes. Distillation supplies an efficient starting point, but reliable AR generation ultimately requires training on the student's own prefix distribution.

4. Making it longer

Moving from a five-second demo to a multi-minute stream is not just a larger version of the same task. The failure mode changes. Small color shifts, pose errors, or object substitutions become part of the conditioning context and can compound over hundreds of blocks. Meanwhile, the KV cache grows, old frames lose influence, and a new prompt can conflict with cached states computed under the old prompt.

Although the papers package it differently, LongLive, Rolling Forcing, and Self-Forcing++ share the same move: stop training only on short, clean clips. The student first builds a much longer AR rollout from its own history; training then advances or samples bounded windows along that rollout. This is the practical meaning of streaming long tuning: train-long-test-long without backpropagating through an entire minute at once.

One long-training idea, three different emphases

5. Distilling it faster

Few-step models are cheap to run only after they have been distilled. The distillation itself can be unusually expensive. A DMD training step may colocate the trainable generator, a frozen real-score model, and a trainable fake-score model. Long self-rollouts add KV cache and activation memory on top. For large video DiTs, the training bill can become the bottleneck that prevents the fast model from existing.

LongLive-2.0: optimize the whole train-to-serve path

LongLive-2.0 treats precision, parallelism, and the AR objective as one infrastructure problem:

• Balanced sequence parallelism assigns clean-history and noisy-target latents from the same temporal chunk to each rank, matching the teacher-forcing mask while balancing loss-bearing tokens.
• NVFP4 training reduces activation and weight memory and accelerates the GEMMs that dominate as video length grows.
• LoRA-only DMD freezes the quantized backbone and trains a small adapter that converts the AR model from four to two steps.
• Deployment matches training through W4A4 execution, quantized KV, SP inference, and VAE/DiT overlap.

The paper reports up to 2.15× training speedup, 1.84× inference speedup, 45.7 FPS on GB200 for its two-step 5B model, and a 0.69× DMD peak-memory ratio with NVFP4 plus LoRA. These numbers should not be compared directly with CausVid or LongLive-1.0 because model size, hardware, resolution, step count, and end-to-end accounting differ. The durable result is the recipe: distillation precision, adapter scope, cache format, and serving kernel should be designed together.

6. Conclusion

The important shift is larger than making diffusion AR. AR turns video generation into a persistent systems interface: past output becomes stable state, new blocks can be streamed incrementally, control can enter at explicit boundaries, and inference infrastructure can manage the session over time.

Distillation made that transition practical for the first generation of systems. CausVid transferred a strong bidirectional prior into a fast AR student; Self Forcing aligned training with the student's own history; LongLive, Rolling Forcing, and Self-Forcing++ extended that history toward minutes; LongLive-2.0 then made the training and serving path itself part of the design.

• Quality under extreme compression. One-step and two-step models still trade motion richness, sharpness, and diversity in ways that current benchmarks only partially capture.
• Long-horizon evaluation. Short-window metrics can miss identity drift, temporal inconsistency, and control failures that appear minutes later.
• AR training beyond conversion. Distillation is effective, but it remains open when training AR video foundations directly will become more robust or economical.
• Multi-request serving. Persistent video sessions combine large KV state, heterogeneous resolutions, different step budgets, and VAE work. Continuous batching for this regime is still immature.
• Cache correctness under interaction. Prompt switches, shot boundaries, RoPE offsets, noisy-context caching, and eviction policies can silently change the effective conditioning distribution.