No AI summary available for this article.
Why It Matters
Gaps remain in our understanding of how large language models (LLMs) acquire knowledge during pre-training.
Provenance
Discovered via ArXiv and published by ArXiv.
Key Claims
Original description
Gaps remain in our understanding of how large language models (LLMs) acquire knowledge during pre-training. We posit that auxiliary views, reformulations of knowledge, are causally helpful for learning. We design controlled experiments to isolate this. First, we confirm that repetition is necessary for acquisition and clarify that paraphrasing helps only at smaller batch sizes. Second, holding the token budget fixed, allocating tokens from document repetition to auxiliary views improves learning, counterintuitively, even for factual recall. Third, the effectiveness of auxiliary views is not co...
Discovered via ArXiv
Research papers and preprints from arXiv.
Publisher: arxiv.org
ID: http://arxiv.org/abs/2609.04180v1 · Indexed about 2 hours ago