AIR schedule targets effective learning rate, not nominal rate
On 128 billion-token MoE pretraining, AIR lowers terminal training loss by about 0.01 versus cosine at matched cumulative learning rate.
Stories per day · 30 days
1 stories, newest first · one card per story
Stories tagged #weight-normalization