DeepMem advances byte-level language modeling with multiscale architecture
Original title: Going Deeper With Memories
DeepMem separates semantic compression from exact byte preservation using a learned invertible subband transform to create a hierarchy of shorter sequences.
Byte level language models provide universal coverage but are expensive. DeepMem separates semantic compression from exact byte preservation using a learned invertible subband transform that constructs a hierarchy of shorter sequences. A lossless carrier and explicit detail sidebands preserve byte identity. Fast weight memory blocks operate along the encoder with iterative shared state computation at the compressed bottleneck while an attention based super resolution decoder restores byte resolution representations. DeepMem achieves stronger language modeling performance than a matched byte level Transformer on FineWeb-Edu and PG19 and obtains the best macro average performance across the evaluated FLORES languages.
- Bottleneck refinement requires approximately 7,600 times fewer counted FLOPs than equivalent full resolution reasoning.
- Improves PIQA accuracy while remaining competitive on code modeling and multiple choice reasoning.
- AI models
- DeepMem
From the original article
1 min read at huggingface.co
Byte level language models avoid tokenizer dependence and provide universal coverage, but processing every byte at full resolution is expensive. We introduce DeepMem, a causal multiscale architecture that separates semantic compression from exact byte preservation.
By Afshin Khadangi · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.