MiniMax Sparse Attention: New Method Enables Efficient Processing of Million-Token Contexts in Large Language Models
Researchers introduced MiniMax Sparse Attention (MSA), a technique that reduces the computational cost of processing extremely long text sequences in large language models by up to 28.4x. The method uses a two-branch architecture to selectively attend to the most relevant portions of text rather than processing all tokens equally. This advancement addresses a critical bottleneck in deploying frontier LLMs for tasks requiring million-token context windows.
MiniMax Sparse Attention is a blockwise sparse attention mechanism built on Grouped Query Attention that enables large language models to efficiently process ultra-long contexts spanning hundreds of thousands to millions of tokens. The system uses a lightweight Index Branch to score and select the most relevant key-value blocks for each attention group, while a Main Branch performs exact attention only over selected blocks. The researchers co-designed the method with specialized GPU execution paths using exp-free Top-k selection and KV-outer sparse attention to maximize tensor-core utilization. Testing on a 109-billion-parameter multimodal model showed MSA achieved 28.4x reduction in per-token attention compute at 1M context length while maintaining performance parity with standard Grouped Query Attention, translating to 14.2x speedups during prefill and 7.6x speedups during decoding on H800 GPUs. The authors have released both the inference kernel and a production-grade multimodal model implementing MSA.
What's missing
The paper does not discuss potential limitations of sparse attention approaches, such as cases where important context may be missed by the selection mechanism, or comparative analysis against other sparse attention methods beyond standard GQA. The generalization of MSA performance across different model sizes, architectures, and downstream tasks remains unclear from the abstract.
What different sources said
- arXiv cs.AICenter
MiniMax Sparse Attention
Related
Topology-Aware Thermodynamics Improves DNA Probe Specificity Design
Researchers developed a new framework for designing DNA probes that accounts for the spatial organization of matched sequences, not just overall thermodynamic stability. Traditional methods rely on scalar measures like melting temperature and free energy, which miss how mismatches are distributed along the probe. The approach could improve diagnostic accuracy in applications like HPV detection and gene expression profiling.
Study Identifies Optimal Thermal Dose for Combining Focused Ultrasound with Immunotherapy in Tumors
Researchers used multimodal PET imaging to identify an optimal thermal dose range for focused ultrasound ablation that destroys tumor tissue while preserving conditions for immunotherapy delivery. The study found that excessive heating collapses blood vessels needed for antibody access, while insufficient heating fails to adequately reduce tumor burden. The findings could guide clinical design of combination treatments pairing thermal ablation with immunotherapies.
Plant MSH1 Protein Functions as Mismatch-Directed Nuclease for Organelle Genome Maintenance
Researchers have identified the precise mechanism by which the AtMSH1 protein in Arabidopsis plants recognizes and cleaves DNA mismatches and lesions, preventing mutations in organellar genomes. The protein combines a DNA mismatch recognition module with a nuclease domain that makes staggered cuts at specific positions relative to DNA damage. This discovery explains how plants maintain unusually low mutation rates in their mitochondrial and chloroplast DNA compared to other eukaryotes.