Can Language Models Play the Wikipedia Game?
GPT-4o, o3-mini, QwQ-32B, and DeepSeek-R1-70B matched humans at Wikipedia-link navigation without clearly beating them.
Researchers used the Wikipedia Game, reaching a target article through links only, to test language-model navigation over collections too large to index. They compared sentence embeddings and LLMs across five Wikipedia languages and five academic-literature datasets, building local graphs from Wikipedia and OpenAlex citations. No model significantly outperformed humans or TF-IDF overall. GPT-o3 mini, GPT-4o, QwQ-32B, and DeepSeek-R1-70B reached comparable success, with rankings changing by language and topic.
- Wikipedia Game used as a large-graph retrieval benchmark
- Local Wikipedia and OpenAlex citation graphs avoid third-party APIs
- Five languages and five academic datasets were tested
- Best models matched humans but did not significantly beat them
From the original article
1 min read at arxiv.orgLanguage models increasingly support or automate information retrieval from document collections too large to index. This task is difficult when it requires specialist linguistic or scientific knowledge. We used the Wikipedia Game to test language models on an analogous information retrieval task.
By Alexander Hickey · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.