No AI summary available for this article.
Why It Matters
Scaling laws predict how loss decreases with increases in computation.
Provenance
Discovered via ArXiv and published by ArXiv.
Key Claims
Original description
Scaling laws predict how loss decreases with increases in computation. We show, contrary to conventional wisdom, that architectural interventions can modify scaling exponents in pre-training, leading to exponential improvements in performance with increases in computation. As an anchoring point, we consider the architectural formulation of looped transformers. Although not typically used in this way, looping, also known as recursive depth, provides a mechanism for model growth, by increasing the number of loops during training. Model growth, with and without shared weights, provides the bigges...
Discovered via ArXiv
Research papers and preprints from arXiv.
Publisher: arxiv.org
ID: http://arxiv.org/abs/2609.19107v1 · Indexed about 1 hour ago