Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data
Study finds Mixture-of-Experts models overfit faster than dense Transformers under repeated training data, with degradation tied to total parameter sparsity.
1 stories, newest first · one card per story
Study finds Mixture-of-Experts models overfit faster than dense Transformers under repeated training data, with degradation tied to total parameter sparsity.
Today 1.6K readers · 2.5K views · 779 reads · all time 189K views