AdaSplash-2 Explained: Entmax Thresholds, On-Chip Histograms and Real GPU Speedups

Why can sparse attention still be slow? A deep mathematical walkthrough of AdaSplash-2: entmax normalization, the threshold tau, left-edge histograms in on-chip SRAM, weighted prefix solvers, safeguarded refinement, packed block masks, and sparse backward computation. Source: Nuno M. T. Gonçalves, Hugo Pitorro, Vlad Niculae, Edoardo M. Ponti, Lei Li, André F. T. Martins, Marcos V. Treviso. AdaSplash-2: Faster Differentiable Sparse Attention. arXiv:2604.15180v1. https://arxiv.org/abs/2604.15180v1 PDF including appendices: https://arxiv.org/pdf/2604.15180v1 Important scope: typical 1–2 iterations means threshold refinement, not one-pass attention or guaranteed exact convergence on every input. Runtime benchmarks use one NVIDIA A6000; model training uses four H100s. Figure 4 measures attention forward plus backward at 4,096–131,072 tokens using sparsity from a selected high-sparsity layer, not whole-model training time. At 131,072 tokens and 85% block sparsity: 1.5 s versus 2.4 s for CUDA FlashAttention-2; rounded plotted labels. No FlashAttention-3/Hopper superiority is established. Element sparsity is not block sparsity. Language-model gains depend on positional controls: NAPE plus entmax is strongest; RoPE counterexamples are included. HELMET results cover its in-context-learning subset, not all tasks. Training/accuracy evaluation extends to 32K; 128K attention measurements do not establish 128K downstream quality. Conversion requires continued training. The video flags endpoint, capacity, and memory-ratio caveats in the v1 appendix. Local worked examples are illustrations, not reproduced GPU benchmarks. This is AdaSplash-2, a separate paper from DashAttention. Ponti is one of the coauthors. Vote for the next paper: https://forms.gle/QLFzswLemfpFx5CXA