Recognizing TTT as linear attention is not merely theoretical — it yields concrete practical benefits. This perspective reveals that many commonly adopted design choices (per-token learning rates, weight normalization, deeper MLPs) are not essential, enabling substantial simplification. Moreover, it makes explicit that the seemingly recurrent inner-loop updates admit a fully parallel formulation, leading to significant efficiency gains. We progressively ablate each component below:
Ablation: Reducing TTT to Linear Attention. Inference throughput (tokens/sec) in recurrent and parallel forms. Restricting updates to the last layer (Variant 1) yields the best perplexity. Removing weight normalization (Variant 2) unlocks parallelization. The full reduction to linear attention (Variant 6) achieves up to 29× throughput over the baseline. Hover for details.