Nexus: Higher-Order Attention Mechanisms in Transformers
Hanting Chen, Chong Zhu, Kai Han, Yuchuan Tian, Yuchen Liang, Tianyu Guo, Xinghao Chen, Dacheng Tao, Yunhe Wang
Abstract
Transformers have achieved significant success across various domains, relying on self-attention to capture dependencies. However, the standard first-order attention mechanism is often limited by a low-rank bottleneck, struggling to capture intricate, multi-hop relationships within a single layer. In this paper, we propose the Nexus, a novel architecture designed to enhance representational power through a recursive framework. Unlike standard approaches that use static linear projections for Queries and Keys, Nexus dynamically refines these representations via nested self-attention mechanisms. Specifically, the Query and Key vectors are themselves outputs of inner attention loops, allowing tokens to aggregate global context and model high-order correlations \textit{prior} to the final attention computation. We enforce a parameter-efficient weight-sharing strategy across recursive steps, ensuring that this enhanced expressivity incurs $\mathcal{O}(1)$ additional parameters. We provide theoretical analysis demonstrating that our method breaks the linear bottleneck of standard attention. Empirically, Nexus outperforms standard Transformers on multiple benchmarks.
Make this part of your paper trail
Save this paper to a shelf, write a review, and keep your own notes.
Community Ratings
★4(1 rating)
4
★★★★★
1 rating
5
0
4
1
3
0
2
0
1
0
Community Reviews
Andrew Peterson
12/8/2025
The paper showcases a recursive attention mechanism which allows it to capture higher-order patterns, i.e. relations between 3 or more tokens. I think the paper is good, but I would've really liked it if they compared to 2-simplical attention. That paper was really interesting because it showed the scaling exponent in the scaling law getting better with their technique, and if this paper had the same result that would've been compelling.