ttok 1.0
Simon Willison releases ttok 1.0, switching the default tokenizer from GPT-4 to the GPT-5 family.
Simon Willison released ttok 1.0, a command-line token counter, after version 0.4 still defaulted to the GPT-4 tokenizer. The 1.0 default uses the GPT-5 family tokenizer, which he also applies to GPT-6. OpenAI has not officially confirmed that GPT-6 shares that tokenizer, but an experiment by William Liu found seven GPT 5.x and 6 models each reported 44,794 tokens and matched on all 31 fixtures.
- ttok 1.0 defaults away from the GPT-4 tokenizer
- OpenAI has not confirmed GPT-6 shares the GPT-5 tokenizer
- Seven GPT 5.x and 6 models matched 44,794 tokens on 31 fixtures
Release: ttok 1.0 I released ttok 0.4 , ran uv tool upgrade ttok , piped a file into the new version... and realized that it was defaulting to the GPT-4 tokenizer when it should very clearly default to GPT-5/GPT-6 instead! I figured switching the default was a reasonable excuse to finally ship a 1.0. OpenAI haven't actually confirmed that GPT-6 uses the same tokenizer as the GPT-5 family yet - there's an angry issue about it - but I found this commit by William Liu which reports on an experiment he ran confirming that the tokenizers are likely the same: All seven GPT models (5.5, 5.6 Sol/Terra/Luna, 6 Astra/Sol/Luna) report 44,794 tokens and match each other on every one of the 31 fixtures.…
This source does not provide full text. Read it at simonwillison.net.