Continuous diffusion language models at two scales
Overview. Nemotron-Labs-Sigma is a family of 3B and 8B continuous diffusion language models for reasoning and coding. Sigma generates text in blocks by denoising compact continuous token embeddings. This smooth ODE trajectory enables inference-time steering and few-step generation, as well as self-speculative decoding. Across reasoning and coding benchmarks, Sigma is competitive with leading masked diffusion and autoregressive baselines and achieves superior performance on code completion.
Learned continuous token embeddings
8 tokens per forward pass on math and 4 on coding, with negligible accuracy loss
Average 8B self-speculative decoding speedup over autoregressive decoding on GB200
Large Language Continuous Diffusion Models
1 NVIDIA 2 Cornell University 3 Georgia Institute of Technology
* Core contributors, equal contribution. † Work done during internship at NVIDIA. ⋊ Equal supervision.
Correspondence: mmardani@nvidia.com
From Gaussian noise to discrete token
Sigma generates text one block at a time, iteratively denoising a compact representation conditioned on the preceding context before reading out tokens.
Initialized from a pretrained autoregressive backbone, Sigma learns its embedding geometry and denoiser together. At inference, classifier-free guidance and score temperature steer the trajectory to balance quality and diversity.

Reasoning, coding, and efficient generation
Strong task performance
Sigma improves over matched baselines across reasoning and coding evaluations, with particularly consistent gains on coding. After supervised fine-tuning, its average performance is on par with Nemotron-Labs-Diffusion at both scales.
Fewer denoising steps
Distribution matching distills the model for few-step generation. The distilled models generate 8 tokens per forward pass on math and 4 on coding, with negligible accuracy loss and preserved diversity.
Superior pass@k performance
Embedding-space steering offers control over the quality–diversity trade-off. Diverse samples continue to improve the chance of finding a correct answer as the sample budget grows.

One model, its own drafter
Sigma shares a backbone between diffusion and autoregressive streams. Its distilled diffusion model can draft blocks for the autoregressive stream to verify, reproducing greedy autoregressive output. On an NVIDIA GB200, the 8B model achieves a 3.46 times average throughput speedup over its autoregressive decoding.

Technical details
Training the model
Sigma trains an autoregressive stream and a diffusion stream on a shared backbone. The former processes clean tokens; the latter denoises their continuous embeddings, attending to the clean prefix and within each block.
Each block trains either token reconstruction or embedding denoising. Training starts from pretrained autoregressive weights and includes an autoregressive loss.

Steering inference
For each block, Sigma iteratively denoises Gaussian noise. In embedding space, classifier-free guidance combines predictions with and without context, while score temperature adjusts the trajectory, steering the trajectory towards the largest mode.
Self-conditioning feeds the previous conditional prediction into the next step. Conditional and unconditional predictions share one forward pass; after denoising, the model reads out tokens and moves to the next block.

Citation
If you find this work useful, please cite our paper as follows:
@article{yang2026large,
title={Large Language Continuous Diffusion Models},
author={Zhihan Yang and Wei Guo and Jean-Marie Lemercier and Simon Welker and Mohammad Mahdi Kamani and Yonggan Fu and Sajad Norouzi and Julius Berner and Tomas Geffner and Karsten Kreis and Yongxin Chen and Molei Tao and John Thickstun and Pavlo Molchanov and Ante Jukić and Arash Vahdat and Morteza Mardani},
journal={arXiv preprint arXiv:2610.02665},
year={2026}
}