No AI summary available for this article.
Why It Matters
LLM serving systems already reuse KV caches, but only when the reused text sits at the very start of the prompt.
Provenance
Discovered via ArXiv and published by ArXiv.
Key Claims
Original description
LLM serving systems already reuse KV caches, but only when the reused text sits at the very start of the prompt. Two growing workloads break this condition: a retrieval-augmented generation server assembles a different set of retrieved chunks for every query, and a multi-agent coordinator reads reports written by other agents. Reused inside a new prompt, a cache carries the wrong positions and never attended to the other sources. The cache may also have been written by a different checkpoint of the same model family, which changes the stored values. Repair methods for such caches have appeared...
Discovered via ArXiv
Research papers and preprints from arXiv.
Publisher: arxiv.org
ID: http://arxiv.org/abs/2609.10266v1 · Indexed about 2 hours ago