GraphDecisionBench: Benchmarking System One Models on Graph Tasks
GraphDecide benchmarks Jev and other choice-based models on graph decision tasks.
GraphDecide is a model-independent benchmark that combines structural task profiles, matched graph-text input contrasts, and heuristic-proposal controls. The authors evaluate Jev and related choice-based models against language-model baselines across fourteen model-interface configurations. Accurate adjacency recognition does not guarantee broader structural correctness, joint graph-text input does not consistently help, and feasible construction does not imply high solution quality. Code and aggregate results are released on GitHub.
- GraphDecide evaluates fourteen model-interface configurations on graph tasks.
- Jev adjacency accuracy does not imply broader structural correctness.
- Joint graph-text input does not consistently improve predictions.
- Feasible graph construction does not establish high solution quality.
From the original article
1 min read at huggingface.co
Large language models (LLMs) are increasingly explored for graph understanding and decision-making, while System One models such as Jev select directly from supplied options. However, the capabilities of System One models on graph-related tasks remain unclear.
By Xianliang Yang, Yapu Zhang, Li Zhao · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.