dNaViT encodes images as discrete general-purpose visual tokens
Authors say it matches or beats task-specific baselines on 12 understanding, 4 generation, and one editing benchmark.
Stories per day · 30 days
4 RSS
Relateddata-protection3pci2buyer-guide1byte-pair-encoding1diffusion1dinov21discrete-codes1encryption1evaluation1fid1format-preserving-encryption1hallucination1image-generation1imagenet1inference-cost1language-models1latex1llava1
20 stories, newest first · one card per story
Stories tagged #tokenization
AdaBPE adapts BPE merge thresholds for mixed language and LaTeX
On five LaTeX and natural-language datasets, the method improves encoding and domain-specific model performance without hurting general language.
Buyer’s guide compares 12 tokenization products for 2026
The roundup sorts vaulted, vaultless, proxy, and SDK products by PCI scope reduction, privacy vaults, and analytics use.
3
Op-ed says encryption fails once sensitive data is in use
An FDD lab assessment found unauthorized restoration of vaultless pseudonyms failed and was logged.
1
Specializing SLM vocabularies can cut inference cost by 77%
A Hugging Face post claims vocabulary specialization cut small-model inference cost by 77% in real applications; no further details were provided.
RubatoCodec adds budgeted adaptive speech tokens and streaming decode
Code and weights are public; adaptive allocation beat fixed-rate coding on reconstruction and intelligibility at low event rates.
Token readout choice confounds POPE VLM hallucination scores
A two-token yes/no logit lookup never matches LLaVA-1.5-7B outputs, shifting adversarial F1 by 6.13 points.
Top 10 Best Tokenization Solutions in 2026 [Ranked & Scored]
A 2026 ranking puts Protegrity, Very Good Security, and Thales atop tokenization platforms.
5
Generative Representation Learning for Scalable Latent Tokenization
A generative tokenizer avoids representation collapse and reaches FID-50K 2.18 with JiT-L.
ttok 0.4
Simon Willison released ttok 0.4, a CLI token counter built on OpenAI's tiktoken library.
BudgetPix: Compute-Adaptive Tokenization for Pixel-Space Image Diffusion
BudgetPix adapts image-diffusion token budgets to visual complexity, matching larger models at a fraction of compute.
Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliability
Cross-tokenizer on-policy distillation works better with strict token alignment than broader, less reliable supervision.
Do Language Models Need a Trainable Input Embedding Table? Fixed Minimal Token Codes at 1.7B-Class Scale
Fixed 16-bit token codes let 1.7B decoder-only models learn substantial language ability without trainable input embeddings.
Who Gets a Token, and What Does It Carry? Unequal Name Support and Concept Access in Large Language Models
A study finds unequal name tokenization in LLMs predicts demographic gaps in task-relevant concept access.
FragToken: Amplifying LLM Inference Costs through Noncanonical Token Generation
FragToken trains LLMs to emit longer non-canonical tokens, roughly doubling inference cost while keeping answers usable.
The Tokens Remember: When Tokenization Bypasses Knowledge Editing and Unlearning
Toketive shows alternative tokenizations can bypass LLM knowledge edits and recover suppressed answers.
The Functionalizer: Lossless Functional Decomposition for Subword Tokenization
The Functionalizer losslessly factors casing, diacritics and character repetition into reversible opcode prefixes, cutting tokenizer vocabulary needs by up to 19.7 percent.
Objective vs. Search: Decomposing What Makes a Good Tokeniser
New tokeniser study shows search procedure, not optimisation objective, drives bits-per-byte performance across model sizes, vocabulary sizes, and multilingual settings.
ActionPiece: Rethinking Action Tokenization for Autoregressive Vision-Language-Action Models
ActionPiece improves action tokenization for vision-language-action models via physical rank consistency, reaching 94.8% on LIBERO with Qwen3-VL-4B.
Rare Not Random Using Token Efficiency for Secrets Scanning
Researcher proposes token efficiency (string length divided by BPE token count) as a better post-regex filter than entropy for secrets scanning, validated on CredData.
2