Octrees as an Explicit 3D Language
OctLLM feeds 3D shapes to an LLM as sparse octree tokens via frozen-backbone trainable branches, cutting image-to-3D FID 17.4% versus ShapeLLM-Omni.
OctLLM represents geometry as explicit sparse octree (S-Octree) occupancy tokens, preserving spatial structure that latent codebooks or coordinate text remove. 3D capacity is added through independent trainable branches in a subset of blocks while text and image tokens keep the frozen vision-language pathway, avoiding full fine-tuning and LoRA. It cuts image-to-3D FID by 17.4% and raises render-grounded captioning by 28.7 points over ShapeLLM-Omni while matching the backbone on general language benchmarks.
- Sparse Octree (S-Octree) tokens preserve spatial structure while shortening full octree sequences
- Independent trainable branches add 3D capacity without modifying pretrained language pathways
- Cuts image-to-3D FID 17.4% and gains 28.7 captioning points over ShapeLLM-Omni
- Matches frozen backbone on language benchmarks while training far fewer parameters than full fine-tuning
- Organizations
- OctLLMShapeLLM-Omni
From the original article
1 min read at huggingface.co
Existing 3D large language models (LLMs) compromise on two fronts: they compress shapes into latent codebook indices or coordinate text, which removes spatial structure from what the model observes, and they acquire the 3D modality by fine-tuning the backbone, which overwrites its general language ability. We present OctLLM, which addresses both limitations.
By Ran Dan, Si-Tong Wei, Pengfei Xiong · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.