Nemotron-Labs-Sigma

Large Language Continuous Diffusion Models

Continuous diffusion for scalable, steerable, and efficient language generation.

Overview. Nemotron-Labs-Sigma is a family of 3B and 8B continuous diffusion language models for reasoning and coding. Sigma generates text in blocks by denoising compact continuous token embeddings. This smooth ODE trajectory enables inference-time steering and few-step generation, as well as self-speculative decoding. Across reasoning and coding benchmarks, Sigma is competitive with leading masked diffusion and autoregressive baselines and achieves superior performance on code completion.

3B / 8B

Continuous diffusion language models at two scales

16D

Learned continuous token embeddings

8 / 4 TPF

8 tokens per forward pass on math and 4 on coding, with negligible accuracy loss

3.46×

Average 8B self-speculative decoding speedup over autoregressive decoding on GB200

Large Language Continuous Diffusion Models

Zhihan Yang*,†,1,2, Wei Guo*,†,1,3, Jean-Marie Lemercier*,1, Simon Welker*,1, Mohammad Mahdi Kamani*,1, Yonggan Fu1, Sajad Norouzi1, Julius Berner1, Tomas Geffner1, Karsten Kreis1, Yongxin Chen3, Molei Tao3, John Thickstun2, Pavlo Molchanov1, Ante Jukić1, Arash Vahdat⋊,1, Morteza Mardani⋊,1

1 NVIDIA   2 Cornell University   3 Georgia Institute of Technology

* Core contributors, equal contribution.   † Work done during internship at NVIDIA.   ⋊ Equal supervision.

Correspondence: mmardani@nvidia.com

From Gaussian noise to discrete token

Sigma generates text one block at a time, iteratively denoising a compact representation conditioned on the preceding context before reading out tokens.

Initialized from a pretrained autoregressive backbone, Sigma learns its embedding geometry and denoiser together. At inference, classifier-free guidance and score temperature steer the trajectory to balance quality and diversity.

Sigma's blockwise continuous diffusion inference pipeline
Sigma refines continuous token representations (16-D) conditioned on a prefix, then commits discrete tokens.

Reasoning, coding, and efficient generation

Strong task performance

Sigma improves over matched baselines across reasoning and coding evaluations, with particularly consistent gains on coding. After supervised fine-tuning, its average performance is on par with Nemotron-Labs-Diffusion at both scales.

Fewer denoising steps

Distribution matching distills the model for few-step generation. The distilled models generate 8 tokens per forward pass on math and 4 on coding, with negligible accuracy loss and preserved diversity.

Superior pass@k performance

Embedding-space steering offers control over the quality–diversity trade-off. Diverse samples continue to improve the chance of finding a correct answer as the sample budget grows.

Paper figure comparing few-step GSM8K performance and scaling with the number of samples on MBPP
Few-step generation on GSM8K and the effect of increasing the sample budget on MBPP.

One model, its own drafter

Sigma shares a backbone between diffusion and autoregressive streams. Its distilled diffusion model can draft blocks for the autoregressive stream to verify, reproducing greedy autoregressive output. On an NVIDIA GB200, the 8B model achieves a 3.46 times average throughput speedup over its autoregressive decoding.

Paper figure showing self-speculative decoding throughput and accepted draft lengths at 3B and 8B
Self-speculative decoding throughput and accepted draft length at 3B and 8B.

Technical details

Training the model

Sigma trains an autoregressive stream and a diffusion stream on a shared backbone. The former processes clean tokens; the latter denoises their continuous embeddings, attending to the clean prefix and within each block.

Each block trains either token reconstruction or embedding denoising. Training starts from pretrained autoregressive weights and includes an autoregressive loss.

Sigma training architecture with autoregressive and diffusion streams
Training combines a clean-token autoregressive stream (left) with a continuous diffusion stream (right).

Steering inference

For each block, Sigma iteratively denoises Gaussian noise. In embedding space, classifier-free guidance combines predictions with and without context, while score temperature adjusts the trajectory, steering the trajectory towards the largest mode.

Self-conditioning feeds the previous conditional prediction into the next step. Conditional and unconditional predictions share one forward pass; after denoising, the model reads out tokens and moves to the next block.

One reverse diffusion step with classifier-free guidance, score temperature, and self-conditioning
One denoising step combines conditional and unconditional predictions with inference-time steering; self-conditioning reuses the previous conditional prediction.

Citation

If you find this work useful, please cite our paper as follows:

@article{yang2026large,
  title={Large Language Continuous Diffusion Models},
  author={Zhihan Yang and Wei Guo and Jean-Marie Lemercier and Simon Welker and Mohammad Mahdi Kamani and Yonggan Fu and Sajad Norouzi and Julius Berner and Tomas Geffner and Karsten Kreis and Yongxin Chen and Molei Tao and John Thickstun and Pavlo Molchanov and Ante Jukić and Arash Vahdat and Morteza Mardani},
  journal={arXiv preprint arXiv:2610.02665},
  year={2026}
}