No AI summary available for this article.
Why It Matters
Two dominant tokenisation algorithms are used by modern language models: byte-pair encoding (BPE) and UnigramLM.
Provenance
Discovered via ArXiv and published by ArXiv.
Key Claims
Original description
Two dominant tokenisation algorithms are used by modern language models: byte-pair encoding (BPE) and UnigramLM. These differ along two orthogonal axes: their optimisation objective (compression vs. log-likelihood) and their search procedure (bottom-up merging vs. top-down pruning). Existing comparisons confound these axes, making it unclear whether their observed differences stem from what is being optimised vs. how it is being optimised. We disentangle the two by introducing two new tokenisation algorithms that complete this 2x2 design space: BottomUpLL, a bottom-up likelihood-based tokenise...
Discovered via ArXiv
Research papers and preprints from arXiv.
Publisher: arxiv.org
ID: http://arxiv.org/abs/2609.19145v1 · Indexed about 1 hour ago