No AI summary available for this article.
Why It Matters
Choosing the rank of a low-rank adaptation (LoRA) update is usually an empirical task.
Provenance
Discovered via ArXiv and published by ArXiv.
Key Claims
Original description
Choosing the rank of a low-rank adaptation (LoRA) update is usually an empirical task. In this paper, we provide a task-dependent theory of the approximation error achievable at each LoRA rank for Transformer attention. We fix a pretrained attention head, a target attention function, and a distribution over inputs from the downstream task, and bound the smallest expected Kullback--Leibler (KL) error achievable by a rank-$r$ query LoRA update. When target attention probabilities are bounded away from zero, we prove a lower bound of the error proportional to $ψ(\|d\|_2)$, where $d$ is the differ...
Discovered via ArXiv
Research papers and preprints from arXiv.
Publisher: arxiv.org
ID: http://arxiv.org/abs/2608.26052v1 · Indexed 4 days ago