Objective vs. Search: Decomposing What Makes a Good Tokeniser
New tokeniser study shows search procedure, not optimisation objective, drives bits-per-byte performance across model sizes, vocabulary sizes, and multilingual settings.
1 stories, newest first · one card per story
New tokeniser study shows search procedure, not optimisation objective, drives bits-per-byte performance across model sizes, vocabulary sizes, and multilingual settings.
Today 1.5K readers · 2.4K views · 757 reads · all time 189K views