titleLLM Explainability with Counterfactual Chains and Causal Graphs
authorsNirit Nussbaum-Hoffer, Nitay Calderon, Liat Ein-Dor, Roi Reichart
date2026-06-04
venuearXiv (preprint, not peer reviewed)
idarXiv:2606.05972
affiliationTechnion Israel institute of technology
upvotes19 (Hugging Face Daily Papers, 2026-11-01)
linksabs pdf hf
abstractCausal graphs provide a high-level language for making mechanisms transparent. Recent work uses Large Language Models (LLMs) to recover causal graphs of external-world processes. Instead, in this paper, we use causal graphs to model LLM inference itself, providing stakeholders with a transparent view of how the model perceives and organizes high-level concepts to produce a prediction. We propose a four-phase method for constructing such graphs. Given a target LLM and a set of textual examples, our method discovers class-discriminative, human-interpretable concepts and maps each input to LLM-perceived concept states. We then introduce an MCMC-inspired counterfactual augmentation procedure that expands the sparse observational data through chains of counterfactuals. This enables stable causal discovery with σ-CG, yielding informative, interpretable graphs. We apply our method to three LLMs across disease diagnosis, sentiment analysis, and LLM-as-a-judge classification tasks. We evaluate the learned graphs for predictive fidelity and structural stability, and the MCMC-inspired augmentation for convergence and downstream utility. Our results show that the discovered causal graphs capture meaningful dependencies consistent with LLMs' reasoning. Together, this paper provides a foundation for concept-level explainability of LLMs.

cite

@misc{nussbaumhoffer2026llm,
  title={LLM Explainability with Counterfactual Chains and Causal Graphs},
  author={Nirit Nussbaum-Hoffer and Nitay Calderon and Liat Ein-Dor and Roi Reichart},
  year={2026},
  eprint={2606.05972},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2606.05972}
}