DGP boosts MLLM performance via dynamic gated pathways
Original title: Where to Connect? Boosting MLLMs via Dynamic Gated Pathways across ALL ViT and LLM Layers
Dynamic Gated Pathways connect all ViT and LLM layers to improve multimodal understanding in LLaVA-1.5 models.
The method establishes adaptive all-to-all connections between every vision encoder layer and every LLM layer using dynamic gating based on semantic state. Built on LLaVA-1.5-7B and 13B, it delivers consistent gains on benchmarks like SQA and MM-Vet. Layer-wise analyses confirm different layers select different visual granularities.
- Consistent gains over LLaVA-1.5-7B baseline.
- Surpasses prior multi-layer fusion methods on most benchmarks.
- Code available at GitHub repository.
- Dynamic routing supports both effectiveness and interpretability.
- Gains observed on SQA^I, MM-Vet, and VizWiz.
- AI models
- LLaVA-1.5
From the original article
1 min read at huggingface.co
A central yet under-explored question in Multimodal Large Language Models (MLLMs) is where to connect---how visual information from a hierarchical vision encoder should be wired into the layer-wise semantics of a large language model.
By Yingying Yan, Jiaqi Tang, Wei Wei · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.