Copy the Same, Distill the Difference: Initializing Linear Vision Transformers
Copying MLP weights and distilling attention lets linear Vision Transformers match or beat pretrained Softmax ViTs.
Stories per day · 30 days
3 RSS
2 stories, newest first · one card per story
Stories tagged #weight-transfer
Agora: Git as Shared Memory for Collective AutoResearch
Agora records multi-agent research as an append-only Git DAG; 13 LLM workers ran nearly 12 days on a weight-transfer problem.