Latent Reasoning: The Next Frontier for LLM Capabilities
Published on 2025-02-10 by AI Research Desk
Chain-of-thought prompting transformed what large language models can accomplish, but it comes with a fundamental limitation: it forces all reasoning into the narrow bottleneck of natural language tokens. Each token carries roughly 15 bits of information, while the model's hidden state at each layer encodes around 40,960 bits. Reasoning in token space wastes the vast majority of the model's representational capacity.
A growing body of research explores whether LLMs can reason directly in their latent representations, bypassing the token bottleneck entirely. Two main paradigms have emerged, each with distinct architectural implications and trade-offs.
Vertical Recurrence: Thinking Deeper
The first paradigm, often called vertical recurrence or activation-based latent reasoning, adds computational depth by reusing transformer layers multiple times on the same input. Instead of passing through each layer once, the model loops through a subset of layers repeatedly, effectively creating a much deeper network without additional parameters.
Universal Transformers pioneered this idea, and more recent work on looped transformers shows that models trained with layer sharing can generalize to harder problems at inference time simply by increasing the number of loop iterations. A model trained on problems requiring 20 reasoning steps can solve problems requiring 40 steps by doubling its loop count, something standard transformers cannot do.
Horizontal Recurrence: Thinking with Hidden State
The second paradigm, horizontal recurrence or hidden-state-based reasoning, introduces special "thinking" tokens into the sequence. These tokens are processed by the model but carry no predefined semantic meaning. The model learns to use them as scratchpad space, encoding intermediate reasoning steps in the hidden states associated with these tokens.
Pause tokens are the simplest example. Appending a fixed number of meaningless tokens (such as periods or the literal string "pause") to the input before generating an answer improves accuracy on reasoning benchmarks. The tokens are semantically empty, yet they give the model additional forward passes in which to refine its internal representations before committing to an output.
The Pre/Loop/Coda Architecture
Several research groups have converged on a three-phase architecture: a pre-processing block that encodes the input, a looped block that iterates for a variable number of steps, and a coda block that decodes the final answer. This Pre/Loop/Coda structure unifies both paradigms. The loop block can implement vertical recurrence through layer reuse, while thinking tokens fed into the loop enable horizontal recurrence.
Perhaps the most intriguing finding is that reasoning depth and reasoning time appear to be interchangeable. Adding more loop iterations and adding more thinking tokens both improve performance on the same tasks, suggesting they tap into the same underlying mechanism: giving the model more computation to resolve ambiguity in its internal representations.
If these approaches scale, the implications are significant. Models could perform complex reasoning without generating visible chain-of-thought, reducing latency and bandwidth costs while potentially surpassing the reasoning quality achievable through token-level thinking alone.
Questions about the datasets referenced in this article? Contact research@securityblueprints.io.