CANOPY: Adaptive-Granularity Evidence Compression for Multimodal RAG
CANOPY compresses multimodal RAG evidence at mixed granularities, cutting reader tokens without hurting answer accuracy.
CANOPY compresses retrieved text, tables, images, and video after multimodal retrieval by representing items as hierarchies and scoring regions with a node encoder fine-tuned on gold evidence. Parent-relative refinement selects multiple regions at different granularities without LLM calls for node pruning, and a critic can request targeted follow-up retrieval when evidence looks insufficient. Across five QA benchmarks on a 33-million-item corpus, it beats the evaluated retrieval baselines, with extra retrieval driving most multi-hop gains. Using unrouted Qwen3-VL-8B-Instruct, compression reduces reader-input tokens by 14.2-27.7% versus the same iterative pipeline without compression, with comparable accuracy.
- A hierarchy node encoder scores evidence regions against the query.
- Parent-relative selection avoids LLM calls for node-level pruning.
- A critic requests follow-up retrieval when accumulated evidence is insufficient.
- On Qwen3-VL-8B-Instruct, reader tokens fall 14.2-27.7% with similar accuracy.
Full article205 words · extracted from huggingface.co · click to collapse
Multimodal RAG retrieves text, tables, images, and videos, but choosing a retrieval granularity does not determine how much context to retain within each item. Coarse units include irrelevant content, while uniformly fine selection can remove context needed to interpret the evidence. Existing compressors address this trade-off with modality-specific mechanisms, leaving open a shared procedure for adapting the retained extent region by region across heterogeneous items. We introduce CANOPY (Canonical Projection over Hierarchy), a framework for adaptive-granularity post-retrieval evidence compression. CANOPY represents retrieved items as hierarchies and uses a node encoder fine-tuned on gold evidence to score regions against the query. Parent-relative refinement compares these scores to select multiple regions at different granularities without LLM calls for node-level pruning. Because compression cannot recover evidence that was never retrieved, a critic requests targeted follow-up retrieval when it judges the accumulated evidence insufficient; newly retrieved items are compressed before being added. Across five QA benchmarks over a 33M-item heterogeneous corpus, CANOPY achieves higher average answer accuracy than the evaluated retrieval baselines. Ablations indicate that additional retrieval drives the main accuracy gains on multi-hop QA. In the unrouted Qwen3-VL-8B-Instruct setting, compression reduces reader-input evidence tokens by 14.2-27.7% relative to the same iterative pipeline without compression, with comparable answer accuracy.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2610.00923