Computation, Not Memory: Chain-of-Thought and State Tracking in LLM Agents
Does an LLM agent's reasoning actually drive its actions? In a task where agents must track a changing hidden state, we separate generating reasoning at each step from keeping earlier reasoning in context. Most of the benefit comes from reasoning written at the moment of action, and a state that probes can decode from activations is a limited guide to what the agent does.
Reasoning helps as step-by-step computation, not as memory