Ariadne transfers fine-tuning effects across differently shaped Transformers
Original title: From Parameter Rebasin to Functional Transfer Across Heterogeneous Transformers
The training-free method synthesizes a native target update from activation changes instead of moving parameters.
Ariadne is a training-free method for reusing task-specific Transformer updates across models with different depths and widths. It does not resize the source model or transport parameters. It matches activation changes at blocks, aligns them with local Procrustes maps, and builds a native target update by closed-form ridge regression. The authors report better target performance and lower time and memory than prior cross-architecture rebasin methods on expansion, reduction, and language-model transfer.
Unconfirmed: Performance and efficiency gains are reported by the paper’s authors only.
- The object transferred is activation change, not the source task vector.
- Alignment uses local Procrustes maps, then ridge regression.
- Evaluations cover Transformer expansion, reduction, and language-model transfer.
- Authors say construction uses less time and memory than prior rebasin methods.
From the original article
1 min read at arxiv.orgAdapting pretrained Transformers to downstream tasks produces task-specific updates that are expensive to reproduce for every model variant. Reusing these updates across different depths and widths is challenging because their parameters have incompatible shapes and do not share a canonical coordinate system.
By Filippo Rinaldi · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.