Flattening Every Memory Peak in Long-Context Mixture-of-Experts Training
Four scheduling techniques bound all memory peaks in long-context MoE training, enabling 120B-667B models at 1M-token context with up to 10.4x throughput.
1 stories, newest first · one card per story
Four scheduling techniques bound all memory peaks in long-context MoE training, enabling 120B-667B models at 1M-token context with up to 10.4x throughput.
Today 1.6K readers · 2.5K views · 779 reads · all time 189K views