On the Token Collapse Phenomenon in Textual Graph Soft Prompting

Andrea Giuseppe Di Francesco; Rishabh Jain; Franco Maria Nardini; Nicola Tonellotto; Pietro Liò; Fabrizio Silvestri

title

authors

Andrea Giuseppe Di Francesco, Rishabh Jain, Franco Maria Nardini, Nicola Tonellotto, Pietro Liò, Fabrizio Silvestri

date

2026-10-22

venue

arXiv (preprint, not peer reviewed)

arXiv:2610.31572

affiliation

Sapienza University of Rome

upvotes

3 (Hugging Face Daily Papers, 2026-11-01)

links

abs pdf hf code

abstract

Recent methods for injecting graph structure into LLM prompts include graph soft prompting, in which a graph neural network (GNN) learns graph-specific tokens that capture relational information among entities in textual prompts. G-Retriever combines graph soft prompts with a verbalized description of the same graph to strengthen the textual prompt signal. We conduct an in-depth analysis of G-Retriever and uncover a pathological behavior we term token collapse. Across three benchmarks, three graph encoders, two LLM families, and both frozen and LoRA-adapted models, G-Retriever's graph tokens become almost indistinguishable across inputs. We provide evidence that the same failure is inherited by subsequent methods, even with multi-token pooling. Under the standard setup, swapping a token with the dataset mean, a token from another example, or one derived from a corrupted graph produces no resolved answer-level effect in accuracy. Moreover, a learned constant prompt that never accesses the graph matches or surpasses G-Retriever on all three benchmarks. These findings indicate that, rather than conveying graph content as intended, the collapsed token functions primarily as a generic soft prompt. We also examine conditions linked to this collapse and propose a two-stage procedure that yields non-collapsed tokens while maintaining competitive performance in most cases. However, token variability alone does not prove that the model uses graph information: via interventions and decoding probes, we show that depending on the configuration, a token can reflect the question, an answer prior, or graph-derived content that the LLM may or may not use. Overall, our results offer a practical approach to evaluating graph-token alignment rather than inferring it from task accuracy alone. Code to reproduce our experiments is available at https://github.com/difra100/Token_Collapse_KGQA.

@misc{francesco2026token,
  title={On the Token Collapse Phenomenon in Textual Graph Soft Prompting},
  author={Andrea Giuseppe Di Francesco and Rishabh Jain and Franco Maria Nardini and Nicola Tonellotto and Pietro Liò and Fabrizio Silvestri},
  year={2026},
  eprint={2610.31572},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2610.31572}
}