Test-time training (TTT) broadly refers to methods that continue to update model parameters during inference. More recently, TTT has been developed as a sequence modeling architecture with linear complexity, serving as an alternative to softmax attention in transformers.

There are two variants of TTT for sequence modeling:

1. TTT-KVB (Sun et al., 2024; Zhang et al., 2025) uses a key-value binding loss (e.g., MSE) as the inner-loop objective — this is our focus.
2. TTT-E2E (Tandon et al., 2025; Behrouz et al., 2025) performs end-to-end backpropagation through the inner loop from the final task loss (e.g., cross-entropy in language modeling).

TTT-KVB works as follows: Each layer maintains fast weights $f_\theta$ (typically a small MLP) updated during both training and inference. Tokens are projected into keys $k$, values $v$, and queries $q$. The model performs two operations to derive the final output:

1. Update: The model performs online gradient descent on $f_\theta$ using a key-value binding loss: $\mathcal{L} = \|f_\theta(k) - v\|^2$.
2. Apply: After updating, the query $q$ is passed through the updated $f_\theta$ to produce the output $o = f_\theta(q)$.

This is commonly interpreted as "memorizing" key-value pairs for later retrieval — an interpretation we challenge below.