Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free

Zihan Qiu∗1, Zekun Wang∗1, Bo Zheng∗1, Zeyu Huang∗2, Kaiyue Wen3, Songlin Yang4, Rui Men1, Le Yu1, Fei Huang1, Suozhi Huang5, Dayiheng LiuB1, Jingren Zhou1, Junyang Li

openreview.net·Read paper

Make this part of your paper trail

Save this paper to a shelf, write a review, and keep your own notes.

Community Reviews

No reviews yet. Be the first to review this paper!