If TTT truly functions by explicitly learning and retrieving key-value associations, its behavior should conform to basic principles of memory formation and optimization dynamics. Instead, we identify systematic anomalies that directly contradict this hypothesis:
Increasing inner-loop iterations improves inner-loop loss but degrades task performance. This inverse relationship directly contradicts the memorization view—if TTT were memorizing key-value pairs, better fitting should improve downstream results.
More inner-loop iterations improve inner-loop loss but degrade task performance.
Replacing gradient descent with gradient ascent preserves—and sometimes even improves—task performance, despite explicitly worsening the key-value fitting objective. This is impossible under a memorization interpretation.
| Model Variant | Perplexity ↓ (LLM) | PSNR ↑ (NVS) | Top-1 Acc ↑ (Image) |
|---|---|---|---|
| Baseline | 16.43 | 25.94 | 79.34% |
| Gradient Ascent | 16.19 | 25.85 | 79.61% |
Queries and keys exhibit significant distributional mismatch in trained models, meaning the inner-loop function is evaluated out-of-distribution. For retrieval to work, Q and K should be in the same semantic space—but they're not.
t-SNE visualization showing Q/K distributional mismatch—the inner loop is evaluated out-of-distribution.
Substituting queries with keys has negligible impact on performance, suggesting queries don't play a functional retrieval role as in standard attention.
| Model Variant | Perplexity ↓ (LLM) | PSNR ↑ (NVS) | Top-1 Acc ↑ (Image) |
|---|---|---|---|
| Baseline | 16.43 | 25.94 | 79.34% |
| Replace Q with K | 16.18 | 25.95 | 79.18% |