Objective vs. Search: Decomposing What Makes a Good Tokeniser
New tokeniser study shows search procedure, not optimisation objective, drives bits-per-byte performance across model sizes, vocabulary sizes, and multilingual settings.
1 stories, newest first · one card per story
New tokeniser study shows search procedure, not optimisation objective, drives bits-per-byte performance across model sizes, vocabulary sizes, and multilingual settings.
Today 1.6K readers · 2.5K views · 779 reads · all time 189K views