Researchers report token collapse in G-Retriever graph soft prompts
Original title: On the Token Collapse Phenomenon in Textual Graph Soft Prompting
A constant prompt that never sees the graph matches or beats G-Retriever on all three benchmarks.
An analysis of G-Retriever finds its graph soft-prompt tokens become nearly indistinguishable across inputs, a failure the authors call token collapse. Across three benchmarks, three graph encoders, two LLM families, and frozen and LoRA-adapted models, swapping tokens produced noresolved answer-level accuracy effect, and a learned constant prompt matched or surpassed G-Retriever. The authors propose a two-stage procedure for non-collapsed tokens and caution that token variability alone does not show the model uses graph content. Code is at github.com/difra100/Token_Collapse_KGQA.
Unconfirmed: Whether the LLM actually uses graph-derived token content depends on configuration and was not settled by accuracy alone.
- Collapse held across three benchmarks, three encoders and two LLM families.
- A dataset-mean, cross-example or corrupted-graph token did not change accuracy.
- Later methods inherited the failure even with multi-token pooling.
- A two-stage procedure yields non-collapsed tokens in most cases.
From the original article
1 min read at huggingface.co
Recent methods for injecting graph structure into LLM prompts include graph soft prompting, in which a graph neural network (GNN) learns graph-specific tokens that capture relational information among entities in textual prompts. G-Retriever combines graph soft prompts with a verbalized description of the same graph to strengthen the textual prompt signal.
By Andrea Giuseppe Di Francesco, Rishabh Jain, Franco Maria Nardini · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.