No AI summary available for this article.
Why It Matters
Gating the value pathway of attention reportedly improves language model pretraining, and prior studies disagree on why.
Provenance
Discovered via ArXiv and published by ArXiv.
Key Claims
Original description
Gating the value pathway of attention reportedly improves language model pretraining, and prior studies disagree on why. We argue and provide experimental evidence that such gates supply two different things that softmax attention lacks: abstention and noise filtering. The first is abstention, which allows an attention head to output nothing, bypassing the requirement that attention weights must sum to one. The second is noise filtering, which allows the value pathway of an attention head to suppress interference from superposed features in the residual stream. In our experiments in matched mo...
Discovered via ArXiv
Research papers and preprints from arXiv.
Publisher: arxiv.org
ID: http://arxiv.org/abs/2609.22005v1 · Indexed about 5 hours ago