Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention

Paper: https://arxiv.org/abs/2502.11089 Notes: https://drive.google.com/open?id=1HLE... 00:00 Intro 01:30 Sparse attention 05:48 Token compression attention 13:10 Token selection attention 20:50 Window attention and putting everything together 28:10 Token selection kernel 34:22 Results