Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention
Complex KDA extends Kimi Delta Attention so one transition can represent 2D rotations and stronger state tracking.
The paper shows Kimi Delta Attention can realize 2D rotations by pairing one delta-rule update with a channel-wise reflection, using gates in [-1,1] and beta in [0,2]. Complex KDA keeps diagonal-plus-rank-one, non-expansive transitions while matching DeltaProduct_2 state-tracking expressivity, including finite subgroups of SO(3) in a single layer. It reports the strongest length extrapolation among tested KDA ranges on S3, S4, and periodic audio, and language-modeling results similar to KDA while beating Transformers and other linear RNNs. OpenEuroLLM released the code and models.
- CKDA allows gates in [-1,1] and delta coefficient beta in [0,2].
- Proves every orthogonal diagonal-plus-rank-one matrix is a CKDA transition.
- One layer tracks finite groups isomorphic to subgroups of SO(3).
- Best tested length extrapolation on S3, S4, and periodic audio.
- Language modeling matches KDA and beats Transformers and other linear RNNs.
Full article242 words · extracted from huggingface.co · click to collapse
Linear RNNs based on the delta-rule enable efficient sequence modeling, but their linear updates with a low-rank correction constrain their expressivity. Prior work has shown that composing two delta-rule transitions in a single recurrent update can model a 2D rotation, but this increases the rank and the cost of the updates compared to a single transition. We show that Kimi Delta Attention (KDA) can realize 2D rotations by combining a single delta-rule transformation with a second reflection supplied by its channel-wise gate. This requires extending the parameter ranges of KDA by combining two existing range extensions: allowing gates in [-1,1] and the delta-rule coefficient β in [0,2]. We call the resulting model Complex KDA (CKDA). It preserves KDA's stability and efficiency, with transitions that remain diagonal-plus-rank-one and non-expansive, while reaching the state-tracking expressivity of DeltaProduct_2. We characterize the expressivity of CKDA and prove that every orthogonal diagonal-plus-rank-one matrix is exactly a CKDA transition matrix. A single CKDA layer can track every finite group isomorphic to a subgroup of SO(3), and many state-tracking results use one fewer layer for CKDA compared to other diagonal-plus-rank-one Linear RNNs. Empirically, combining both extensions yields the strongest length extrapolation among tested KDA range settings on S_3, S_4, and periodic audio continuation. In language modeling, CKDA outperforms Transformers and other linear RNNs, obtains similar results to a KDA baseline, and shows promising scaling behavior. Our code is open source at https://github.com/OpenEuroLLM/ComplexKDA and our models are available at https://huggingface.co/collections/openeurollm/complexkda.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.24797