Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention
Complex KDA extends Kimi Delta Attention so one transition can represent 2D rotations and stronger state tracking.
The paper shows Kimi Delta Attention can realize 2D rotations by pairing one delta-rule update with a channel-wise reflection, using gates in [-1,1] and beta in [0,2]. Complex KDA keeps diagonal-plus-rank-one, non-expansive transitions while matching DeltaProduct_2 state-tracking expressivity, including finite subgroups of SO(3) in a single layer. It reports the strongest length extrapolation among tested KDA ranges on S3, S4, and periodic audio, and language-modeling results similar to KDA while beating Transformers and other linear RNNs. OpenEuroLLM released the code and models.
- CKDA allows gates in [-1,1] and delta coefficient beta in [0,2].
- Proves every orthogonal diagonal-plus-rank-one matrix is a CKDA transition.
- One layer tracks finite groups isomorphic to subgroups of SO(3).
- Best tested length extrapolation on S3, S4, and periodic audio.
- Language modeling matches KDA and beats Transformers and other linear RNNs.
- Organizations
- OpenEuroLLM
- AI models
- Complex KDAKimi Delta Attention
From the original article
1 min read at huggingface.co
Linear RNNs based on the delta-rule enable efficient sequence modeling, but their linear updates with a low-rank correction constrain their expressivity. Prior work has shown that composing two delta-rule transitions in a single recurrent update can model a 2D rotation, but this increases the rank and the cost of the updates compared to a single transition.
By Julien Siems, Riccardo Grazzi, Korbinian Pöppel · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.