ttok 1.0
Simon Willison releases ttok 1.0, switching the default tokenizer from GPT-4 to the GPT-5 family.
Stories per day · 30 days
2 RSS
6 stories, newest first · one card per story
Stories tagged #tokenizer
Introducing Claude Haiku 5.5
Anthropic launched Claude Haiku 5.5, matching GPT-6 Luna at $0.10 input and $0.50 output per million tokens.
SemanTok: Predictable Semantic Tokens for Efficient Autoregressive Video Generation
SemanTok aligns video tokens with frozen DINO features so short prefixes carry clip semantics.
DC-SAE: Deep Compression Semantic Autoencoder for Faster Diffusion Convergence
DC-SAE compresses images 32x and speeds diffusion training while beating DC-AE on ImageNet.
tokenizers v1: encode, decode and scaling, measured
Hugging Face releases tokenizers v1 with improved encode/decode operations and scaling benchmarks.
Mothersuperior/yue2-mothersuperior-realaudio-tokenizer-v4 — new model trending #30 on Hugging Face
Community release ships a real-audio semantic tokenizer and NAR LoRA for YuE2-3B, enabling artist-style song generation and covers from user recordings.