A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training

Zihan Qiu, Zeyu Huang, Kaiyue Wen, Peng Jin, Bo Zheng, Yuxin Zhou, Haofeng Huang, Zekun Wang, Xiao Li, Huaqing Zhang, Yang Xu, Haoran Lian, Siqi Zhang, Rui Men, Jianwei Zhang, Ivan Titov, Dayiheng Liu, Jingren Zhou, Junyang Lin

arXiv1/30/2026·Read paper

Abstract

Researchers from the Qwen Team, University of Edinburgh, Stanford, and Tsinghua University reframe emergent outliers in Large Language Models, like attention and residual sinks, as essential...

Make this part of your paper trail

Save this paper to a shelf, write a review, and keep your own notes.

Community Reviews

No reviews yet. Be the first to review this paper!