LoopVQ enables sub 1-bit KV cache quantization for looped transformers
LoopVQ quantizes KV cache at 0.5 bits per element for looped transformers, nearly halving decode latency at 32K context.
Stories per day · 30 days
Relateddynamic-layer-ordering1kv-cache-quantization1llm-inference1looped-transformers1memory-optimization1multilingual-nlp1runtime-composition1
3 stories, newest first · one card per story
Stories tagged #transformer-architecture
Library of Layers dynamically orders transformer layers at runtime
LoL uses a single hyperparameter for its global layer library to generalize dense, recurrent, and mixture-of-experts architectures with distinct ordering for different tasks.
A Princeton Researcher Proposes Recurrent Looped Transformer (RLT) that Carries Decoder State across Every Token, Fixing 96 Blocks per Token with Unbounded Temporal Depth
Princeton researcher Yifan Zhang proposes Recurrent Looped Transformer, carrying full decoder state across every token for unbounded temporal depth.
4