Prism ML releases Ternary-Bonsai-2-27B, a 27B ternary-weight model derived from Qwen3.8-27B, running on laptops at 8.6 GB with 98.2% FP16 quality retained.
Prism ML released Ternary-Bonsai-2-27B, a 27.36B-parameter model derived from Qwen3.8-27B with fully ternary {-1, 0, +1} weights in a Hadamard-rotated basis. The MLX 2-bit pack is 8.60 GB on disk versus ~54 GB FP16 and averages 84.78 across 14 thinking-mode benchmarks, claimed to retain 98.2% of FP16 intelligence. It keeps 262K-token context via a ~75% linear-attention backbone, ships custom ternary MLX and CUDA kernels, and has a GGUF companion (PTQ1_0 at 5.95 GB, PQ2_0 at 7.21 GB) for llama.cpp. Licensed Apache 2.0.
8.60 GB on disk vs ~54 GB FP16, roughly a 7x reduction
84.78 average across 14 benchmarks, 98.2% of FP16 quality
End-to-end ternary weights with Hadamard-rotated basis, no FP16 fallback
262K context on-device via ~75% linear-attention Qwen3.8-27B backbone
Full 27B-class reasoning in ternary transformer weights — on everyday laptops
> **8.60 GB** on disk, language model + vision tower | **98.2%** of FP16 intelligence retained | **\~47 tok/s** on an Apple M5 Max laptop
## Highlights
- **8.60 GB** on disk: a 7.67 GB language model (down from \~54 GB FP16) plus the 0.92 GB vision tower. MLX's container stores a scale and a bias per group, so the language model costs 2.25 bits/weight where the same ternary weights take 1.75 in the GGUF PTQ1_0 packing
- **98.2% of FP16 intelligence retained**: 84.78 average across 14 thinking-mode benchmarks — far above the conventional IQ2_XXS build (72.59) at less than two-thirds of its footprint, and within 0.4 points of UD-Q4_K_XL at three times the footprint
- **Retains thinking, reasoning, and agentic behavior** deep in the sub-4-bit regime, where conventional low-bit representations collapse: math within half a point of full precision (96.57), coding level with the baseline (89.42), agentic tool calling at 74.92
- **End-to-end ternary language weights** across embeddings, attention projections, MLP projections, and LM head, with no high-precision escape hatches behind a low-bit label — a *true* 1.72 bits per weight as a representation, 2.25 as MLX stores it; the vision tower is bundled in this pack, unquantized
- **262K-token context** on-device, kept practical by the Qwen3.8-27B hybrid-attention backbone (\~75% linear attention)
- **Custom ternary hybrid-attention kernels** on Apple MLX (Python, Swift) and CUDA — packed weights are consumed directly, never expanded back to FP16
- **GGUF companion**: also available as [Ternary-Bonsai-2-27B-gguf](https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf) for llama.cpp (CUDA, Metal, CPU), in two packings — PTQ1_0 (5.95 GB) and PQ2_0 (7.21 GB)
## Resources
- **[Whitepaper](https://github.com/PrismML-Eng/Bonsai-demo/blob/main/bonsai-2-27b-whitepaper.pdf)** — full methodology, benchmarks, and measurement notes
- **[Demo & examples](https://github.com/PrismML-Eng/Bonsai-demo)** — **the source of truth for running these models**: tested setup for every backend, pinned binaries, serving, benchmarking and integration, kept current as the runtimes move
| Base model | Derived from Qwen3.8-27B, a 27B hybrid-attention causal language model (architecture unchanged) |
| Parameters | 27.36B total — 24.35B language backbone (64 blocks) + 2.54B embedding/LM head + 0.46B vision tower (27 blocks) |
| Architecture | Hybrid attention (\~75% linear / \~25% full attention), SwiGLU MLP, RoPE, RMSNorm |
| Context length | 262K tokens (inherited from the base model; kept practical on-device by the predominantly linear-attention backbone) |
| Weight format | Ternary g128: {−1, 0, +1} weights with FP16 group-wise scaling |
| Weight basis | Blockwise Hadamard rotation (block 1024, fixed ±1 signs) folded into the stored weights; the matching transform is applied to activations at runtime |
| Vision tower | included in this pack, 0.92 GB FP16, the official Qwen3.8-27B tower, unquantized |
| Deployed size | **8.60 GB** MLX safetensors on disk: 7.67 GB language model + 0.92 GB vision tower |
| Backends | Apple MLX (Python, Swift) and CUDA |
| License | Apache 2.0 |
## Weight Representation: Ternary g128
Each weight takes a value from {−1, 0, +1}, with one shared FP16 scale factor for every group of 128 weights. A ternary value carries log₂3 ≈ 1.585 bits of information, so the effective storage cost of the format is **\~1.71 bits/weight** (ternary code + 16-bit scale amortized over 128 weights); counting the small set of tensors held above the ternary representation brings the model as a whole to **1.72 bits/weight** — an idealized \~9.3x reduction vs FP16.
The weights are stored in a **rotated basis**: each matrix is transformed blockwise by an orthogonal Hadamard rotation before the ternary assignment, and the runtime applies the matching transform to activations. The rotation is folded into the stored weights offline, so it costs no extra bits and no extra weight traffic; the packed model declares its rotation as metadata, so a runtime either applies the matching transform or refuses to load the file.
Those are language-model figures; the pack on disk is 8.60 GB once the 0.92 GB vision tower is included.
Practical deployment needs packing formats that efficient kernels can consume, and Bonsai 2 ships two: **PTQ1_0** packs trits densely and lands essentially on the information-theoretic target, while **PQ2_0** stores each trit in a 2-bit slot, trading footprint for cheaper unpacking. Neither is uniformly faster — see the throughput table below. These sizes describe the language model alone, the only component that must stay resident for text inference; 26.2M parameters (**0.0976%** of the language model — the recurrent state path of the linear-attention layers, plus the normalization weights) remain in higher precision and are counted in the 1.72 figure.
Unlike conventional low-bit builds — whose advertised labels understate their true average bit-width (a widely-used "2-bit" build of Qwen3.8-27B is really 2.8 bits/weight at 9.4 GB) — the Bonsai representation carries a bit-width that matches its name.
### MLX Packaging
This pack carries **both** the language model and the vision tower. `model.safetensors` is 8.60 GB:
7.67 GB of packed language weights and 0.92 GB for the tower, which is the official Qwen3.8-27B
vision tower carried unrotated and unquantized in FP16. The Hadamard rotation applies only to the
language model's projections, so the tower needs no transform and is plain passthrough.
MLX's grouped low-bit format stores both a scale and a bias per group. The ternary levels
`{-s, 0, +s}` are reproduced exactly by setting `scale = s` and `bias = -s`, so the 2-bit codes
`{0, 1, 2}` decode to `-s`, `0`, `+s`. The bias carries no new information, but the container stores
two FP16 values per group of 128 where the native format stores one. The effective rate is therefore
2.25 bits/weight against PQ2_0's 2.13. This is a container property, not a different representation:
the packed weights decode to exactly the same ternary values as the GGUF bands, verified by comparing
the group scales bit for bit.
## Best Practices
### Generation Parameters
We recommend using the following sets of sampling parameters for generation:
These match the base model's own `generation_config.json` and are the values carried in the GGUF
metadata (`general.sampling.*`), so a client that reads model defaults will use them without being
told. They are also the settings used for the reported benchmark results (thinking mode).
The model uses `xhigh` reasoning effort by default; use `medium` for shorter responses and a balance of speed and accuracy. `low` reasoning effort is not supported and when selected the model will behave close to `xhigh`.
### System Prompt
You can use a simple system prompt such as:
```
You are a helpful assistant
```
## Quickstart
> **[PrismML-Eng/Bonsai-demo](https://github.com/PrismML-Eng/Bonsai-demo) is the source of truth for
> running these models.** It carries the tested setup, builds the right MLX runtime, and is kept
> current. Where anything here disagrees with it, it is right.
Pass no images for text-only use. `chat_config` is needed because mlx-vlm's prompt helper keys off
`model_type`. Everything runs on stock packages, no fork; `PACK-RUNTIME.md` documents the contract
and [Bonsai-demo](https://github.com/PrismML-Eng/Bonsai-demo) carries the tested setup.
For CUDA, CPU, and llama.cpp on Metal, use the GGUF packs of the same weights:
**[Ternary-Bonsai-2-27B-gguf](https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf)**. Those need
a build of our [llama.cpp fork](https://github.com/PrismML-Eng/llama.cpp); stock llama.cpp cannot run
them.
## Cross-Platform Throughput
`tg128` is token-generation throughput over 128 generated tokens (the memory-bandwidth-bound, interactive phase); `pp512` is prompt-processing throughput over 512 input tokens (the compute-bound phase). Both in tokens/s. Rows are measured with llama.cpp (Metal/CUDA, custom low-bit kernels) on the GGUF packs of the same weights, at batch size 1 and depth 0 with no vision tower. NVIDIA energy is board power including HBM/GDDR.
Hacker News · AI