REX moves hazardous knowledge into removable experts in MoE models
Original title: Towards Architectural-Level Safety Control: Relocating Hazardous Knowledge into Removable Experts in Mixture-of-Experts Models
Excising those experts lowers WMDP-Bio by 48.4 points on Qwen3.5-35B-A3B while MMLU changes by less than 3 points.
Researchers propose REX, which adds physically removable experts to Mixture-of-Experts models and concentrates hazardous knowledge there. Expert expansion, selective gradient masking and routing differentiation move that knowledge out of the original experts. Removal held on Intern-S2, Qwen3.5-35B-A3B and Gemma-4-26B-A4B, and unlike nine unlearning baselines it largely persisted after later fine-tuning.
Unconfirmed: The accuracy figures are reported by the paper's authors and are not independently confirmed here.
- After 250M fine-tuning tokens, REX WMDP-Bio rose from 32.4% to 42.8%.
- The strongest baseline rebounded from 44.1% to 78.3% on WMDP-Bio.
- Authors say shared-parameter unlearning suppresses expression rather than removing knowledge.
From the original article
1 min read at arxiv.orgLarge language models acquire broad knowledge from web-scale corpora during pretraining, including knowledge that could be hazardous if misused. Removing such knowledge from the model itself without degrading general capability is desirable but difficult, as it is deeply entangled with general capability within the same parameters.
By Cheng Jin · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.