The initial reaction to reasoning models — o1, o3, DeepSeek-R1, Claude's extended thinking mode — was fair skepticism. Ten to a hundred times the token spend for chain-of-thought the model used to do in a cheaper regime. On raw benchmarks the gains were real but not universal. On production dashboards the cost line was harder to ignore.
Then we started reading the traces.
The undervalued part is the transcript
When an agent picks the wrong tool five percent of the time, a non-reasoning model gives you a wrong tool and a shrug. A reasoning model gives you a paragraph of "I considered call_search first but the query already looked like a customer ID, so I chose lookup_customer" — and now you can see, in plain English, where its heuristic diverged from yours.
That is not a benchmark. That is a debugger.
We now default to running the reasoning tier during eval and development, even for products that will ship on a smaller model in prod. The trace is a source of truth about the prompt: what the model thought the constraints were, which examples it anchored on, where the ambiguity sits.
How we use it in practice
- Bug reports. A failing eval fixture gets re-run with reasoning on. The trace goes straight into the ticket. Root cause is usually a missing constraint the prompt never named.
- Prompt refactors. Before changing a prompt, sample ten traces on the current version. Look for the sentences the model quotes when it's about to make the right decision. Move those closer to the top on the next version.
- Model swaps. Running the same input through two models and diffing their traces beats staring at output-only benchmarks. Where the reasoning diverges is where your prompt is under-specified.
Two failure modes to avoid
Do not trust the trace as ground truth. A reasoning model can produce a coherent explanation for a wrong answer. The trace tells you what the model attended to; it doesn't tell you the model was right to attend to it.
Do not leak the trace to users. Reasoning content will contain names, IDs, and hypotheses the model rejected. That is exactly the class of data you never want in a support reply. Log it, review it internally, strip it before it goes to a customer. Store it in a bucket that can be turned off in an incident.
The literature is catching up
The best 2025 and 2026 papers on tool-use failure — Anthropic's, DeepMind's, several from academic labs — use reasoning traces as the primary artifact. When a paper claims a model "planned poorly," they are looking at a paragraph, not a metric. That's a hint about what your own eval pipeline should be collecting.
The model can explain itself. The question is whether you're listening.