Neural Thickets gains traced to output diversity rather than weight experts
Original title: Rethinking Neural Thickets Without Weight Perturbations
Perturb-Select-Vote on frozen models matches or beats full-weight perturbations on most reasoning tasks and helps vision under domain shift.
The authors reproduce reported Neural Thickets gains but argue they do not require dense neighborhoods of task-specific experts in weight space. Perturbing one decoder block recovers much of the improvement, and a random bias on output logits of a frozen model matches or exceeds full-weight perturbations on most reasoning tasks. They formalize Perturb-Select-Vote, linking the effect to sampling, self-consistency and verifier-guided selection, and report that logit PSV improves vision under domain shift, including Earth observation, while full-weight perturbations add little.
Unconfirmed: The alternative explanation is the authors' interpretation of their reproductions, not an independent confirmation.
- A random logit bias on a frozen model matches or exceeds full-weight perturbations on most reasoning tasks.
- Small weight changes act mainly through induced logit shifts, the authors say.
- Logit PSV keeps the model fully frozen and training-free.
- Vision tests include domain shift and Earth observation.
From the original article
1 min read at arxiv.orgThe Neural Thickets hypothesis posits that pretrained language models lie in dense neighborhoods of task-specific experts in weight space, which random Gaussian perturbations can uncover. We reproduce the reported gains, but show that they do not require this geometric explanation.
By Nicolas Dufour · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.