A Systematic Study of Semantic ID Spaces for Generative Information Retrieval
Study unifies product and residual quantization for semantic DocIDs and tests them on MS MARCO and NQ.
The paper studies what makes effective numerical document identifiers for generative information retrieval, where a model generates DocIDs instead of using retrieve-and-rank. It unifies product quantization, residual quantization, and hybrids in one design space covering hierarchy versus parallelism, DocID length, and codebook size. Training-free intrinsic metrics estimate DocID quality without full model training. Experiments on MS MARCO 300K and NQ320K relate those structural properties to retrieval effectiveness.
- Unifies product quantization, residual quantization, and hybrid DocID designs.
- Introduces training-free intrinsic metrics for DocID quality.
- Studies hierarchy, parallelism, identifier length, and codebook size.
- Experiments use MS MARCO 300K and NQ320K.
From the original article
1 min read at arxiv.orgGenerative Information Retrieval (GIR) has emerged as a transformative paradigm, shifting document retrieval from a traditional "retrieve-and-rank" workflow to sequence-to-sequence generation, where a model directly predicts document identifiers (DocIDs). While the semantic design of these DocIDs is known to be critical for performance, a fundamental question remains under-explored: what makes a good DocID?
By Alexia Allal · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.