Near-zero embeddings stall gated model training
Untied gated checkpoints contain near-zero embeddings that amplify gradients 30+ orders of magnitude and trigger FP32 clipping, freezing rows in 20% of Telugu pretraining steps.
Stories per day · 30 days
3 RSS
Relatedembeddings1flow-matching1gated-deltanet1gated-models1language-models1libero1linear-attention1long-context1merlin1recurrent-models1robotics1test-time-training1training1unlocking1vision-language-action1
8 stories, newest first · one card per story
Stories tagged #mamba
ParaTTT parallelizes deep nonlinear test-time training memories
Two refinement steps at 4K context evaluate the inner-loop trajectory 32× faster than TTT-MLP at matched error.
MERLIN uses key moments to route multi-state linear attention
Reported long-context QA gains are +8.3 and +7.0 points on Gated DeltaNet-1.3B and Mamba-2-780M, with larger LongBench lifts.
MambaAct swaps pi0.5 transformer action expert for compact Mamba
The public model keeps the pi0.5 vision-language encoder, shrinks the action expert 10 times, and raises mean LIBERO success by 1.08 points.
History Replays Itself: Unleashing Long-Context Capability in Linear Attention
NoteBank lifts pretrained linear-attention models to million-token retrieval by replaying sparse episodic memory.
TIDES: Implicit Time-Awareness in Selective State Space Models
TIDES keeps physical time gaps in selective state-space models and tops irregular time-series benchmarks.
Low-Discrepancy Dither for Quantized Recurrent State Caches
Golden-ratio Weyl dither keeps quantized Mamba recurrent states closer to full precision than stochastic rounding.
SLICEChat: Progressive In-Encoder Token Pruning for Whole-Slide Pathology Language Models
SLICEChat prunes pathology slide tokens inside a hybrid Mamba-Transformer encoder before multimodal fusion.