Higher-order pruning of experts in mixture-of-experts language models
New HOPE method uses second-order objectives to prune Mixture-of-Experts LLMs, outperforming REAP at 50% pruning on models up to 122B parameters.
38
New HOPE method uses second-order objectives to prune Mixture-of-Experts LLMs, outperforming REAP at 50% pruning on models up to 122B parameters.
Today 1K readers · 1.5K views · 474 reads · all time 184K views