ReWY speeds up delta-rule attention gradients
ReWY reformulates backward passes for delta-rule linear attention reducing seven local WY matrix products to three and achieving 1.15-1.81x speedup on GPUs.
Stories per day · 30 days
1 stories, newest first · one card per story