G-Retriever graph tokens collapse into a generic soft prompt
Original title: On the Token Collapse Phenomenon in Textual Graph Soft Prompting
A learned constant prompt that never sees the graph matches or beats G-Retriever on all three benchmarks.
An analysis of G-Retriever finds token collapse: graph soft-prompt tokens become nearly indistinguishable across inputs. The pattern held across three benchmarks, three graph encoders, two LLM families, and both frozen and LoRA-adapted models, and later methods inherit it even with multi-token pooling. Swapping a token for the dataset mean, another example, or a corrupted graph did not change answer-level accuracy, while a constant prompt matched or surpassed G-Retriever. A two-stage procedure yields non-collapsed tokens with competitive performance in most cases, but token variability does not prove the model uses graph content.
Unconfirmed: Whether a non-collapsed token is actually used by the LLM depends on configuration and is not settled by accuracy alone.
- Collapse appeared with frozen and LoRA-adapted models in two LLM families.
- A constant prompt never accessing the graph matched or beat G-Retriever.
- Token swaps, including a corrupted-graph token, left accuracy unchanged.
- Probes show a token may reflect the question, an answer prior, or graph content.
- Reproduction code is at github.com/difra100/Token_Collapse_KGQA.
From the original article
1 min read at arxiv.orgRecent methods for injecting graph structure into LLM prompts include graph soft prompting, in which a graph neural network (GNN) learns graph-specific tokens that capture relational information among entities in textual prompts. G-Retriever combines graph soft prompts with a verbalized description of the same graph to strengthen the textual prompt signal.
By Andrea Giuseppe Di Francesco · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.