ArXiv▲ 43

TACO: Ternary Absolute-max Column-wise One-sparse Optimizer for LLM Fine-Tuning

ArXiv|Thursday, October 1, 2026|about 11 hours ago|Jichao Jiang

No AI summary available for this article.

Why It Matters

Full-parameter fine-tuning of large language models (LLMs) incurs substantial optimizer state memory overhead, limiting the model sizes that fit on modern GPUs.

Provenance

Discovered via ArXiv and published by ArXiv.

Open original ↗Source profile →

Key Claims

Existing approaches either compress optimizer state, abandon first-order gradients, or change the update geometry while retaining dense state.
The recently introduced Muon optimizer reduces optimizer memory through matrix-valued updates.
Still, its geometry differs from AdamW and can lead to performance degradation when fine-tuning AdamW-pretrained models.

Original description

Full-parameter fine-tuning of large language models (LLMs) incurs substantial optimizer state memory overhead, limiting the model sizes that fit on modern GPUs. Existing approaches either compress optimizer state, abandon first-order gradients, or change the update geometry while retaining dense state. The recently introduced Muon optimizer reduces optimizer memory through matrix-valued updates. Still, its geometry differs from AdamW and can lead to performance degradation when fine-tuning AdamW-pretrained models. To reduce optimizer memory without sacrificing accuracy or computational efficie...

arxiv cs.LG math.OC publisher:arxiv tier:research

Read on ArXiv ↗← Back to feed

Discovered via ArXiv

Research papers and preprints from arXiv.

Publisher: arxiv.org

ID: http://arxiv.org/abs/2610.02199v1 · Indexed about 1 hour ago