No AI summary available for this article.
Why It Matters
Out-of-distribution length generalization, namely to extrapolate a task from short to longer context, has been studied intensively for transformers.
Provenance
Discovered via ArXiv and published by ArXiv.
Key Claims
Original description
Out-of-distribution length generalization, namely to extrapolate a task from short to longer context, has been studied intensively for transformers. Here we focus on distance generalization, which probes performance when inter-token distances are changed between training and inference, while keeping a fixed context length. We construct two synthetic delay copy tasks, both involving finite distances between source and recall, where tokens are copied either fully or selectively, and test models on delays unseen during training. We address three questions: (A) Do positional encoding schemes such...
Discovered via ArXiv
Research papers and preprints from arXiv.
Publisher: arxiv.org
ID: http://arxiv.org/abs/2609.11913v1 · Indexed about 3 hours ago