Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes It
A rank-8 LoRA lets frozen transformers follow much longer in-context reference chains.
The paper finds pretrained transformers use little of their depth to follow in-context references: thirteen base models reliably follow only 1.4–3.6 lines, and extra pretrained loops add little. A task-trained rank-8 LoRA on one early layer, with all weights frozen, raises Qwen3-8B from 15.5% to 99% exact accuracy on 24-line chains; longer training reaches 50 lines. Ouro-1.4B reaches 60 lines after four loops and at least 160 after eight. The LoRA starts a middle-layer relay that frozen heads read further along, and task-specific LoRAs also improve MuSiQue.
- Thirteen base models reliably follow only 1.4 to 3.6 reference lines.
- Rank-8 LoRA lifts Qwen3-8B from 15.5% to 99% on 24-line chains.
- Ouro-1.4B reaches 60 lines after four loops and 160 after eight.
- The LoRA relay requires parent-line attention across middle layers.
- Task-specific LoRAs also improve MuSiQue multi-hop question answering.
From the original article
1 min read at huggingface.co
Pretrained transformers use little of their depth to follow references in context. Thirteen base models reliably follow only 1. 4-3.
By Zehao Jin, Ruixuan Deng, Junran Wang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.