Computation, Not Memory: Chain-of-Thought and State Tracking in LLM Agents

The task

Moving files while locks change in the background

The agent works in a small file system with three log files and report.txt. It must archive every unlocked log file, never touch a locked file, and leave report.txt locked. After the agent's second, fourth and sixth commands, a background job toggles the lock of one file. The announcement names the file but not its new value, so the agent has to work out the current locks from its history.

A real episode of Qwen3-8B (seed 304), run once with reasoning disabled and once with reasoning enabled. Both runs start from the same state and receive the same toggles. Reasoning excerpts are verbatim and shortened.

Question 1

Generating reasoning matters more than retaining it

Reasoning can help in two ways. Generating it before an action gives the model extra computation at the moment of decision. Retaining it in later prompts gives the model notes that it could consult instead of working out the state again. We vary the two separately. The observations and executed commands always stay in the prompt.

Violation rate: lock-violating commands divided by eligible attempts to move or delete a file. Conditions are named by mode (think or direct) and by what is retained from earlier replies (full, body or cmd). Qwen3-8B: 500 episodes per condition; Qwen3.5-9B: 300. Retention does matter for task completion in Qwen3.5-9B, raising strict success from 53% to 65%.

Question 2

The content of the reasoning matters

Agents that reason also end up in different states from agents that do not. To compare like with like, we take 400 saved prompts and complete each one with a different thought block. Only the thought block changes.

Example from Qwen3-8B (episode 76, turn 3). At this point gamma.log is locked. Reasoning and commands are produced by greedy decoding. Other-history reasoning was written by the same model for a different episode. Qwen3.5-9B shows the same pattern: own reasoning lowers the rate from 40% to 4%.

Question 2, continued

The action follows the reasoning, even against the history

We build pairs of prompts that differ in one fact: the initial lock of one log file. The model writes its own reasoning for each version. We then give each version the reasoning written for the other one.

Example pair from Qwen3-8B (episode 219, turn 2). The table counts pairs in which the agent moves the file. In history A, moving it violates the rule. When the reasoning is held fixed, changing the history changes whether the file is moved in at most two pairs.

Question 3

A state that can be decoded is not necessarily the one that is used

Without reasoning, can we read the state from the model instead? Linear probes recover the lock state from the activations about as well as a text classifier recovers it from the transcript (balanced accuracy 0.79 and 0.77). To test which directions affect the model's answer, we ask "Is the file locked?" on both versions of a counterfactual pair. We then copy the locked version's activation at the final token into the unlocked version, along one direction only. The recovered fraction R says how far this moves the answer: 0 means no change, and 1 means the answer of the locked version.

Qwen3-8B, final token of the state question, 120 held-out counterfactual pairs, layers 20 to 35. Decoding: AUROC on held-out ordinary prompts, with bars drawn from chance (0.5) to perfect (1.0). The logistic probe decodes the state almost perfectly from layer 22 on, but never recovers more than 0.02. R can slightly exceed 1 when a patch moves the answer a little past the locked version's answer, as the counterfactual direction does in the last two layers. On the same prefixes, the state question and the direct action also depend on different parts of the context (Section 7 of the paper).

Takeaways

• In this task, the benefit of reasoning for rule-following comes mainly from generating it at each step, not from retaining it as memory.
• The benefit depends on what the reasoning says, and the agent's next action follows the reasoning in its context.
• An accurate readout of the state, from probes or from the model's answers, does not show how the state reaches the action. Readouts should be tested against the behavior they are meant to explain.

Citation

@misc{mouseli2026computation,
  title  = {Computation, Not Memory: Chain-of-Thought and State Tracking in {LLM} Agents},
  author = {Mouseli, Pedram and Moayedi, Massieh},
  year   = {2026},
  eprint = {2610.25760},
  archivePrefix = {arXiv}
}