Higher-order pruning of experts in mixture-of-experts language models
New HOPE method uses second-order objectives to prune Mixture-of-Experts LLMs, outperforming REAP at 50% pruning on models up to 122B parameters.
1 stories, newest first · one card per story
New HOPE method uses second-order objectives to prune Mixture-of-Experts LLMs, outperforming REAP at 50% pruning on models up to 122B parameters.
Today 1.6K readers · 2.4K views · 775 reads · all time 189K views