FactorEngram: Factorized N-gram Memory with Basis-Level Gating for Language Models
FactorEngram adds factorized n-gram memory with basis-level gating to language models.
Stories per day · 30 days
2 stories, newest first · one card per story
Stories tagged #gating
Abstention and Noise Filtering: Two Missing Primitives of Softmax Attention
Gating softmax attention supplies abstention and noise filtering, with filtering gains growing from 10M to 350M models.