Multi-agent AI team finds linear attention designs that beat GDN2
Original title: Building AI Research Teams for Autonomous Discovery of Linear Attention Architectures
Agents with distinct disciplines, private histories, and adaptive allocation report gains over GDN2 at 1.3B scale.
The paper builds persistent AI researchers grounded in different disciplines who share experimental evidence while keeping private research histories. Adaptive allocation chooses which researchers propose and run experiments under a limited budget. Searching for linear attention architectures, the team reports better language modeling, common-sense reasoning, and retrieval than GDN2 at 1.3B scale. Ablations that drop backgrounds, private histories, or adaptive selection produce weaker discoveries.
Unconfirmed: Performance gains are the authors' reported comparisons, not an independent evaluation.
- Without disciplinary backgrounds, the architecture search plateaus early.
- Private histories improve progress even when shared evidence is identical.
- Uniform selection underperforms adaptive allocation on the same budget.
- Evaluation covers synthetic and real-world retrieval tasks.
From the original article
1 min read at huggingface.co
Autonomous agents can increasingly propose ideas, modify code, and run experiments, creating an opportunity to accelerate scientific discovery using teams of collaborative AI researchers. To escape local minima in the discovery, we propose to encourage orthogonal exploration by constructing the team using agents with varying expertise. However, realizing this design requires more than diverse backgrounds.
By Yujian Liu, Li An, Zelin He · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.