Willison ships ttok 0.4, then 1.0 with GPT-5 default
Simon Willison released ttok 0.4 and then 1.0, switching the default tokenizer from GPT-4 to the GPT-5 family.
Simon Willison released ttok 0.4, a command-line token counter built on OpenAI's tiktoken library and the first update in about two years. That release fixes a Click warning, refreshes continuous integration, adds a --list-models command for tiktoken-supported models, and can be run with uvx without installing the package. The following day he released ttok 1.0 because 0.4 still defaulted to the GPT-4 tokenizer; 1.0 defaults to the GPT-5 family tokenizer, which he also applies to GPT-6. OpenAI has not officially confirmed that GPT-6 shares that tokenizer. An experiment by William Liu found seven GPT 5.x and 6 models each reported 44,794 tokens and matched on all 31 fixtures. The two reports cover successive versions and do not contradict each other.
- Simon Willison released ttok 0.4 (report dated 2026-10-08) and ttok 1.0 (report dated 2026-10-09).
- ttok is a command-line token counter built on OpenAI's open-source tiktoken library.
- Version 0.4 was the first update in about two years; it fixes a Click warning, refreshes continuous integration, and adds a --list-models flag for tiktoken-supported models.
- Users can count tokens with uvx without installing the package, including by piping a file into uvx ttok.
- ttok 1.0 changes the default from the GPT-4 tokenizer, which 0.4 still used, to the GPT-5 family tokenizer, which Willison also applies to GPT-6.
- OpenAI has not officially confirmed that GPT-6 shares the GPT-5 tokenizer.
- An experiment by William Liu found seven GPT 5.x and 6 models each reported 44,794 tokens and matched on all 31 fixtures.