Starting from a real failure

Mechanistic interpretability has produced compelling case studies — induction heads, gender bias circuits, spurious features in probes — but these have mostly been post-hoc analyses of known behaviors on controlled inputs. A harder question is whether the tools can drive safety-relevant predictions: given a model and a class of inputs, can we use circuits to anticipate which cases will go wrong before they do?

Anthropic's 2026 NLA paper proposed a concrete testbed for exactly this, grounded in a real observed failure of Claude Opus: misreported tool calls. The model is shown a tool response containing a wrong answer. In some cases it parrots the tool; in others it ignores it and recovers its own correct answer. The paper identified this as a promising case study for interpretability precisely because the mechanism should, in principle, be visible in the circuit: either the tool value is being read and propagated to the output, or it is not.

We took that proposal and asked: can we operationalise it at scale and evaluate whether attribution graphs actually deliver on that promise?

Designing the testbed

To give the interpretability tools every advantage, we stripped the task down to its minimal form. The model is asked a simple arithmetic question, calls an execute_code tool, and receives a wrong answer. The only question is what the model does next.

The design has no confounders by construction. Prompt structure is identical across all cases; only the tool's returned digit changes. There is no multi-step reasoning, no ambiguous entities, no retrieval. The decision collapses to one token. This is the closest thing to a controlled experiment available in the language model setting.

We generated 480 arithmetic cases (subtraction and addition, single- and multi-digit answers), ran a full alpha sweep on each, and retained the 292 that were fully consistent: all-parrot cases where the model parrots every tested wrong digit, and all-recover cases where it ignores the tool regardless. Mixed cases — where behavior depends on which wrong digit was provided — were excluded. This gives us a ground truth that is robust, not a single-shot sample.

What the circuit oracle found

We built attribution graphs for each case using Gemma-3-4b-it with 262k-wide transcoders, then ran Circuit Oracle — a multi-agent LLM pipeline that navigates the graph, calls get_source_influence to measure the signed influence flowing from the tool-response token to the output logit, inspects feature autointerp labels on Neuronpedia, and emits a verdict. The oracle is blind: the output token is masked to · before it sees the graph, so it cannot read the answer and must derive the mechanism from the circuit alone.

We evaluated on 94 cases drawn from two graph types: raw correct-tool-call graphs (alpha = 0, where the tool is telling the truth), and probe-weighted circuits (graphs attributed toward learned digit-probe directions). A blind judge — a separate LLM given only the oracle's written analysis — classified each report as PARROT or RECOVER without knowing the ground truth.

This result fits a broader pattern. A recent evaluation of interpretability tools by Marks et al. (CHIVE, 2026) found that activation-reading tools — including autoencoders and activation oracles — provided "no uplift" over agents that saw only chat transcripts when predicting whether prompt edits would change model behavior. The interpretability tools described what the model computed, but not whether that computation was causally downstream of the input feature under test. We see the same failure mode here.

The asymmetry between parrot (92%) and recover (51%) is informative. Parrot cases are easy: a digit-detector feature fires on the tool-response token, carries the tool's digit forward, and drives the output. The circuit shows a clear tool → feature → output story, and the signed influence from the tool position is measurable. Recover cases are hard precisely because the same features are present — the correct tool value is in the prompt too — but their influence on the output is inert. A static circuit cannot distinguish "this feature activated and mattered" from "this feature activated but was overridden."

Asking the counterfactual question directly

The parrot/recover distinction is fundamentally a sensitivity question: if the tool value changes, does the output change? Attribution graphs answer a different question — what actually ran — and inferring sensitivity from structure is unreliable.

We formalise this by parameterising the tool-response token with a scalar α. At α = 0 the tool reports the correct answer; at α = 1 it reports a wrong one. We implement the interpolation at the embedding level:

At α = 0 both groups look identical to the model — the tool is correct, so all cases output the correct answer. The groups differ in how quickly diff(α) falls as α increases. A parrot case's logit difference crosses zero at some flip-alpha; a recover case's stays positive all the way to α = 1.

The key observation: the local slope of diff at α = 0 — computed by a single backward pass through the original model — already encodes which group a case belongs to. This is a directional derivative: how sensitive is the model's output to a perturbation of the tool embedding, measured at the correct-tool end? Parrot cases are sensitive; recover cases are not.

Results

We evaluate the Taylor predictor on the full 292-case set, grid-searching the threshold. We compare three variants: the original HF model, the replacement model (with transcoders), and the replacement model with attention patterns and LayerNorm scales frozen at α = 0 — the frozen curve is the circuit's own linear prediction of how the output responds to the tool changing.

Several things are worth noting. The α = 0 logit diff alone is useless — it sits at majority-class accuracy regardless of threshold, because both groups output the correct digit at α = 0 by construction. The value does not separate them; the slope does. This rules out any explanation where the alpha=0 representation directly encodes parrot/recover; what's encoded is sensitivity.

The frozen circuit (77%) outperforms the oracle reading that same circuit (67%), which is telling: the circuit's own linear response to an embedding perturbation predicts better than a sophisticated LLM agent trying to interpret its structure. The agent is looking at the right object but asking the wrong question.

The gap between frozen circuit (77%) and original Taylor (85%) reflects transcoder approximation error — replacing 42 MLP layers with sparse dictionaries introduces small mismatches that compound into a 8-point accuracy gap. The circuit is a good approximation, but not perfect.

Discussion

Our result should not be read as "attribution graphs are useless." They remain the richest mechanistic description of a forward pass available, and the oracle's 92% accuracy on parrot cases shows they can surface clean causal stories when the mechanism is genuinely unambiguous. The limitation is specific: attribution graphs are observational, and behavioral prediction often requires counterfactual reasoning.

The Taylor approach works here because we constructed a meaningful perturbation direction — interpolating between the correct and wrong tool embeddings — and measured sensitivity along it. This required knowing what to perturb, which came from understanding the task structure. In settings where the relevant perturbation direction is not known in advance, the approach would need to be adapted.

A natural next step is to combine both: use attribution graphs to identify which features and positions are most relevant to a behavior, then apply targeted perturbations along those directions to test causal claims. The graph provides the vocabulary; the gradient provides the test. Neither alone is sufficient.