Rethinking LLM-Guided Visual Question Answering: Does Deeper Knowledge Integration Really Help?
Study finds shallow LLM guidance beats deep fusion for knowledge-based VQA under scarce data.
This paper tests how deeply LLM guidance should be integrated into knowledge-based visual question answering. Using Llama 3.3 70B to produce structured reasoning, visual focus, distractors, and external knowledge, the authors compare late fusion with attention-matrix injection inside a frozen ViT and FLAN-T5 system on OK-VQA. Shallow late fusion scores 39.4% exact match, 10.0 points above the no-guidance baseline, while deeper attention modification reaches 31.8% under data-scarce fine-tuning. An LLM paraphrasing pipeline also expands the training set more than tenfold.
- Guidance is generated with Llama 3.3 70B for an OK-VQA study.
- Both strategies use a frozen ViT plus FLAN-T5 framework.
- Late fusion reaches 39.4% exact match, 10 points above no guidance.
- Deep attention-matrix injection scores 31.8% under scarce fine-tuning data.
- Question paraphrasing expands training data more than tenfold.
From the original article
1 min read at huggingface.co
Knowledge-based Visual Question Answering (VQA) requires reasoning that extends beyond visible image content, demanding external commonsense and factual knowledge. While recent work leverages Large Language Models (LLMs) as knowledge sources, the question of how deeply such guidance should be integrated into a vision language architecture remains underexplored.
By Aneerban Saha, Vijay Kumar, Sameer Madhav D. · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.