LLM Inference

Four-bit floating point is moving from a storage-only compression trick to a primitive for training and inference across LLMs, diffusion, video generation, KV cache, and attention.

Why KV cache compression methods that work on paper fail in production — FlashAttention hides attention scores, and repeated token eviction frees no paged GPU memory — and how a geometric property hidden beneath RoPE resolves both.