LiLiCorr: Lightweight Likelihood Correlation of Parallel Drafts for Speculative Decoding

Paper

Matan Rusanovsky, Yoav Miron, Roy Uziel, Omer Belhasin, Ran Zilberstein, Maor Ashkenazi, Michael Elad

TL;DR. A parallel block drafter predicts a whole block of future tokens in one forward pass, but it is trained per position, so the tokens it emits are individually plausible and jointly incoherent. Methods concurrent to ours restore coherence by running a correction network autoregressively, once at every position. LiLiCorr instead keeps the few highest-scoring candidate tokens at every position and scores them all in a single network pass, reducing the correlation logic to computations that parallelize efficiently and leaving only a cheap path selection sequential. Against those concurrent works, LiLiCorr is the fastest system in almost all the scenarios we evaluate.

Overview

Speculative decoding accelerates language-model inference by letting a small drafter propose several future tokens that the target model verifies in parallel, keeping the longest correct prefix. One target pass over the whole block replaces one pass per token, so the speedup rests on two quantities: how many draft tokens survive verification, and what the drafting overhead is. Drafter design is therefore a latency-accuracy trade-off: a more accurate drafter wins longer accepted prefixes, but only pays off if it is cheap enough to run.

Diffusion-style parallel block drafters such as DFlash sit at an attractive point on that trade-off. They predict an entire block of future tokens in a single forward pass, so proposing a long speculative window costs slightly more than drafting a single token, which keeps the drafting overhead low.

An ideal parallel drafter would sample from the target’s exact joint distribution over the tokens in the block. However, that is combinatorially infeasible: a block of B positions over a vocabulary of size V has V^B joint realizations, so the joint distribution can never be materialized, let alone sampled from exactly. One practical approximation is to supervise each position on its own, with a per-position cross-entropy that constrains only the token marginal at that position.

This leaves the joint distribution largely unconstrained. Even if each position is modeled accurately in isolation, assembling them independently can still yield a poor block sample, because it ignores the dependencies among positions. In the example below, “dog” and “meows” are each the most probable token at their own position, yet they do not belong together. Whether “meows” fits at the second position depends on what the first position committed to. Taking each argmax independently yields “The dog meows sweetly”, which the verifier rejects as soon as it reaches the first inconsistent token, in this case “meows”.

Each position scored on its own, then scored again once the first has committed. Left, independent per-position marginals. Right, position 2 scored again once position 1 has committed to “dog”. The ordering flips: “meows” leads on its own marginal and falls to the bottom given “dog”. That dependency is exactly what a per-position cross-entropy does not constrain. Probabilities are illustrative.

Restoring coherence on top of those marginals is a known problem. Each of the existing approaches comes with a trade-off. Diffusion language models such as LLaDA, MDLM, Dream and Nemotron-Labs-Diffusion correlate the sequence by iterative refinement, re-masking and re-predicting subsets of tokens so that later rounds are conditioned on earlier ones. However, in speculative decoding, every round requires an additional draft model pass, which negates the latency benefit that motivated parallel drafting in the first place. DDTree instead expands the marginals into a draft tree, leaving the target to verify more tokens within its single pass. The cost lands on verification, which speculative decoding needs to keep close to a single-token forward pass through the target. A tree gives the target more tokens to verify, under tree attention that is typically less efficient than causal attention. Domino and DSpark, two works concurrent with ours, keep the correction on the drafter side, but reintroduce a sequential dependency between positions by running a correction network once at every position, bringing back the per-position computation that a parallel drafter exists to avoid.

We introduce LiLiCorr, a Lightweight Likelihood-based model that Correlates the per-position marginals a parallel drafter produces. It sits on top of any parallel block drafter, needing only the per-position distributions it emits and the hidden states behind them. LiLiCorr recovers the block’s joint structure from a single cheap network pass over those candidates. The pass produces a pair of vectors per candidate, and every correlation between neighboring positions follows from them directly, as matrix operations, without the joint ever being materialized. What remains is a greedy walk over those precomputed scores, which invokes no learned parameters at all.

How It Works

Score candidates, do not re-predict tokens

Instead of reasoning over the full vocabulary, LiLiCorr keeps the top-K candidate tokens the drafter ranked highest at each position, which across the block form a candidate lattice. The input vector for each candidate is formed from four terms, each projected and summed: the target’s frozen embedding of the candidate token, the drafter’s hidden state at that position, a handful of features read from the drafter’s distribution, which summarize how confident it is in this particular token, and learned positional information, representing the token’s position in the block and its rank among the candidates at that position. A small two-layer Transformer then processes the whole lattice in a single pass, so each candidate comes out represented in the context of the entire block. Its output for each candidate is finally fused with the target’s projected hidden state of the last verified token, so the verifier’s view of the context, and not the drafter’s alone, shapes the score.

From the vocabulary to the candidate lattice, and one pass of attention over it. Left, at each position the drafter’s full distribution is reduced to its top-K tokens, and those candidates are what LiLiCorr reads. Right: the scoring network itself, a two-layer Transformer whose self-attention spans the entire lattice, so every candidate is represented against every other. Its output is then fused with the target’s state at the last verified token.

Correlation as cosine similarities

From the fused output of each candidate, the model then produces two vectors, an out vector and an in vector. Two neighboring candidates fit together when the earlier candidate’s out vector aligns with the later candidate’s in vector. The coupling is pairwise, but the information behind it is not: the vectors are produced from the whole lattice, so the score of a pair is shaped by evidence outside it, among the candidates that come before and after. Keeping the coupling pairwise is what makes the lattice cheap to score, and it means the joint distribution is never materialized.

Two vectors per candidate, one cosine similarity per pair. Each candidate carries an out vector and an in vector. A pair of neighbors is scored by the cosine similarity of the earlier candidate’s out vector with the later candidate’s in vector, so “meows” scores highly after “cat” and its competitors do not.

Training: promote the ground-truth path

Training turns those similarities into likelihoods. At each position the K similarities to the ground-truth predecessor are passed through a softmax. A cross-entropy then raises the likelihood of the ground-truth continuation and pushes the competing ones down, which promotes the ground-truth path through the lattice over the alternatives. Supervision applies only while the ground-truth token is still among the K candidates, since once it drops out of that pool no reranking can bring it back. A second term additionally penalizes the competitors that the target itself scores far below the ground truth. Further details are in the paper.

Training promotes the ground-truth continuation. Conditioned on the ground-truth predecessor, the similarities to the K candidates at a position become a distribution over them, and cross-entropy pulls the ground-truth continuation up and the rest down. The probabilities shown are the softmax of the similarities beside them.

Decoding: similarities are computed in parallel

A block of B positions has B-1 adjacent pairs, and at each pair every candidate is matched against every candidate at the next position, which requires calculating K² cosine similarities. The exception is the first pair, whose predecessor is the single token the target has already verified, so it contributes only K. That is (B-2)K² + K similarities for the whole block, or 904 at the B = 16 and K = 8 we use. All of them are computed in parallel, as one efficient batch of small matrix products.

Every similarity in the block, computed in parallel, and one interior pair as a single matmul. Because the coupling is pairwise and the vectors are already computed, each adjacent pair reduces to a single small matrix product, batched across the block.

A cheap greedy walk

The block is then decoded left to right, each position selecting the candidate that best follows the one selected before it. This requires calculating an argmax over K precomputed scores per position, so no network weights are involved.

One argmax per position, over scores that already exist. Starting from the last verified token, each position takes the candidate that best follows the one selected before it. This is the only sequential step, and it reads scores that already exist.

Results

We evaluate on nine benchmarks: math (GSM8K, MATH-500, AIME 2025), code (HumanEval, MBPP, LiveCodeBench), chat (AlpacaEval, MT-Bench), and the qualitative split of SPEED-Bench, whose prompts span 11 categories reaching beyond those three: multilingual text, summarization and roleplay among them. Each runs at two target sizes under greedy and temperature-one decoding, which gives 36 settings. SPEED-Bench also provides a throughput split, which holds input length fixed and varies serving concurrency. To match the lengths the drafters were trained on we use the 1K and 2K ISLs, at concurrencies 1, 2, 4, 8, 16 and 32, across its three entropy tiers, for 36 more operating points. The tiers group domains by how predictable their text is: coding and math are low entropy and easier to draft, while roleplay and writing are high entropy and harder.

The baselines are vanilla DFlash, Domino and DSpark. LiLiCorr builds on DFlash. All four systems are trained on the same corpus for the same number of epochs and served on the same deployment-representative stack with the same optimization effort, so no drafter is disadvantaged by its implementation or its training corpus.

Speedup over the target’s own autoregressive decoding, per benchmark, for both target sizes under both decoding regimes. Measured on a single H100 at concurrency one. All systems are trained on the same corpus and served on the same stack. LiLiCorr’s value is printed above each group, in green where it leads and in the rival’s colour where a rival is ahead. DSpark has a sampled variant that applies only at temperature one.

Production systems typically serve multiple requests concurrently, so decoding operates at batch sizes greater than one and becomes increasingly compute-bound. Although speedups diminish for all methods as batch size grows, their relative ordering remains consistent.

Speedup over autoregressive decoding against serving concurrency, on the SPEED-Bench throughput split. Qwen3-8B target on a single H100, 512 prompts per block.

• LiLiCorr is the fastest system in almost all evaluated scenarios. It holds the highest throughput in 35 of the 36 benchmark settings above, and in 35 of the 36 operating points of the throughput split, 70 of 72 in total. Under greedy decoding it leads in all 18 settings, by up to 4.9% on the benchmark average over the strongest baseline, and across the concurrency sweep by up to 7.2%.
• LiLiCorr improves the drafter it builds on, on every benchmark. Against a vanilla DFlash drafter trained on the same data, it raises acceptance length by 9 to 19%, and under greedy decoding raises throughput by 4 to 13%.
• The widest margin falls on held-out data. Across SPEED-Bench’s 11 categories, LiLiCorr leads under greedy decoding on every one, and by the largest margin on multilingual, by 11.0% on Qwen3-8B and 16.8% on Qwen3-4B. Multilingual is the one domain excluded from the training corpus, which suggests the correlation generalizes on OOD data.

Generalizing to longer contexts

The same pattern persists on SPEED-Bench’s 8K, 16K and 32K ISL splits, which place every drafter well beyond its training range. We serve these with YaRN extending the positional encoding at inference, applied identically to all systems, since without it every drafter attends over positions outside its training range and gives up acceptance length. With YaRN, LiLiCorr serves faster than every baseline at 46 of the 54 long ISL operating points, by up to 10.8%.

A small change to training closes the remaining gap. We add a hinge, an additional penalty that applies whenever the correct candidate’s lead over its closest competitor is narrower than a fixed margin. This makes the reranker’s decisions more robust at long inputs, and trained this way LiLiCorr holds the highest throughput at every one of the 90 operating points of the throughput split, inside the training range and beyond.

Speedup over autoregressive decoding across every block of the SPEED-Bench throughput split. ISL 1K and 2K sit inside the drafter’s training range, while 8K, 16K and 32K reach an order of magnitude past it. LiLiCorr here is trained with the additional hinge term, and leads at all 90 operating points shown. Qwen3-8B target on a single H100.

Citation

@misc{rusanovsky2026lilicorrlightweightlikelihoodcorrelation,
      title={LiLiCorr: Lightweight Likelihood Correlation of Parallel Drafts for Speculative Decoding}, 
      author={Matan Rusanovsky and Yoav Miron and Roy Uziel and Omer Belhasin and Ran Zilberstein and Maor Ashkenazi and Michael Elad},
      year={2026},
      eprint={2608.20530},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2608.20530}, 
}