When a language model writes down its reasoning, does the evidence it cites actually drive the conclusion it states? That is the question our new preprint asks. The paper is available on Zenodo at DOI 10.5281/zenodo.23308620.
What we did
We ran a pre-registered causal test of declared evidence relations in Scholialang reasoning traces, on Qwen2.5-7B-Instruct, using a controlled comparison task. In Scholialang a model emits typed, machine-checkable traces in which a Finding atom declares its dependence on an Evidence atom. We tested that declared edge with activation patching: we spliced a counterfactual Evidence span's activations into the clean context and checked whether the Finding's read was redirected the way the relation predicts.
What we found
For this model and task, a model's emitted symbolic label can causally steer its subsequent conclusion, even when the accompanying facts disagree.
Flipping only the polarity label (supports/refutes) redirected the conclusion in 73 of 90 relations. Swapping only the numbers never changed the selected status under this intervention, although in a post-hoc analysis it shifted the decision margin toward the counterfactual status in every case. What we establish is downstream label influence.
The metric was validated on cases where task relevance is known by construction. Under the pre-registered protocol it separated relevant from irrelevant evidence perfectly (J = 1.0 in both evidence orders). On two batteries, patching the cited evidence redirected the conclusion in 77/90 and 53/71 relations, roughly 75 to 86 percent, with substantial variance between batteries. Unrelated-evidence controls never redirected.
What it does not show
We have not shown that the method detects genuine unfaithfulness. Behavioral arms on two hosted models were direction-sensitive to their own evidence edits, but an authority-cue arm induced no unfaithfulness for the method to catch. The results are on an open 7B model and synthetic test cases. Where the numeric comparison is computed, and whether the declared reference structure itself controls routing, remain open: we did not intervene on how the label was generated.
Why it matters
Reasoning that software depends on needs to be checkable, not just readable. A trace that declares "this finding depends on that evidence" is making a claim about the model's computation. This paper describes one reproducible way to test such a claim, and reports what we found when we did.
Brewster, D. and Sevig, B. (2026). Does the Evidence a Model Cites Drive Its Conclusion? A Pre-registered Causal Test of Declared Evidence Relations in Structured Reasoning Traces. Zenodo. DOI: 10.5281/zenodo.23308620


