prism-ml/Ternary-Bonsai-2-27B-gguf — new model trending #29 on Hugging Face

<p align="center">

<img src="./assets/bonsai-logo.svg" width="280" alt="Bonsai">

</p>

<p align="center">

<a href="https://prismml.com"><b>Prism ML Website</b></a> &nbsp;|&nbsp;

<a href="https://github.com/PrismML-Eng/Bonsai-demo/blob/main/bonsai-2-27b-whitepaper.pdf"><b>Whitepaper</b></a> &nbsp;|&nbsp;

<a href="https://github.com/PrismML-Eng/Bonsai-demo"><b>Demo &amp; Examples</b></a> &nbsp;|&nbsp;

<a href="https://discord.gg/prismml"><b>Discord</b></a>

</p>

# Bonsai 2 27B — GGUF

Full 27B-class reasoning in ternary transformer weights, for llama.cpp (CUDA, Metal, CPU)

> **\~9.3x** smaller than FP16 (ideal) | **98.2%** of FP16 intelligence retained | **\~47 tok/s** on an Apple M5 Max laptop

## Highlights

- **\~5.9 GB** language model (down from \~54 GB FP16) — full 27B-class reasoning on a standard laptop or a single GPU

- **98.2% of FP16 intelligence retained**: 84.78 average across 14 thinking-mode benchmarks — far above the conventional IQ2_XXS build (72.59) at less than two-thirds of its footprint, and within 0.4 points of UD-Q4_K_XL at three times the footprint

- **Retains thinking, reasoning, and agentic behavior** deep in the sub-4-bit regime, where conventional low-bit representations collapse: math within half a point of full precision (96.57), coding level with the baseline (89.42), agentic tool calling at 74.92

- **End-to-end ternary language weights** across embeddings, attention projections, MLP projections, and LM head, at a *true* 1.72 bits per weight — no high-precision escape hatches behind a low-bit label; the vision tower ships as a separate Q8_0 mmproj pack

- **262K-token context** on-device, kept practical by the Qwen3.8-27B hybrid-attention backbone (\~75% linear attention)

- **Two GGUF packings** with custom ternary hybrid-attention kernels for llama.cpp (CUDA, Metal) — **PTQ1_0** packs trits densely (1.75 bits/weight, 5.95 GB), **PQ2_0** stores each trit in a 2-bit slot (2.13 bits/weight, 7.21 GB); packed weights are consumed directly, never expanded back to FP16

- **MLX companion**: also available as [Ternary-Bonsai-2-27B-mlx-2bit](https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-mlx-2bit) for native Apple Silicon inference

## Resources

- **[Whitepaper](https://github.com/PrismML-Eng/Bonsai-demo/blob/main/bonsai-2-27b-whitepaper.pdf)** — full methodology, benchmarks, and measurement notes

- **[Demo & examples](https://github.com/PrismML-Eng/Bonsai-demo)** — **the source of truth for running these models**: tested setup for every backend, pinned binaries, serving, benchmarking and integration, kept current as the runtimes move

- **Low-bit kernels**: [llama.cpp fork](https://github.com/PrismML-Eng/llama.cpp) (CUDA + Metal) · [MLX fork](https://github.com/PrismML-Eng/mlx) (Apple Silicon) · [mlx-swift fork](https://github.com/PrismML-Eng/mlx-swift) (iOS/macOS)

- **[Discord](https://discord.gg/prismml)** — join the community for support, discussion, and updates

## Model Overview

| Item | Specification |

| :---------------- | :----------------------------------------------------------------------------------------------- |

| Base model | Derived from Qwen3.8-27B, a 27B hybrid-attention causal language model (architecture unchanged) |

| Parameters | 27.36B total — 24.35B language backbone (64 blocks) + 2.54B embedding/LM head + 0.46B vision tower (27 blocks) |

| Architecture | Hybrid attention (\~75% linear / \~25% full attention), SwiGLU MLP, RoPE, RMSNorm |

| Context length | 262K tokens (inherited from the base model; kept practical on-device by the predominantly linear-attention backbone) |

| Weight format | Ternary g128: {−1, 0, +1} weights with FP16 group-wise scaling, packed as **PTQ1_0** (dense trits) or **PQ2_0** (2-bit slots) |

| Weight basis | Blockwise Hadamard rotation (block 1024, fixed ±1 signs) folded into the stored weights; the matching transform is applied to activations at runtime |

| Low-bit coverage | Embeddings, attention projections, MLP projections, LM head |

| Vision tower | optional \~0.63 GB mmproj pack (Q8_0), loaded only for image input |

| Deployed size | **5.95 GB** (PTQ1_0) or **7.21 GB** (PQ2_0); 5.8 GB ideal at 1.72 bits/weight — see below |

| Backends | llama.cpp (CUDA, Metal, CPU) |

| License | Apache 2.0 |

## Weight Representation: Ternary g128

Each weight takes a value from {−1, 0, +1}, with one shared FP16 scale factor for every group of 128 weights. A ternary value carries log₂3 ≈ 1.585 bits of information, so the effective storage cost of the format is **\~1.71 bits/weight** (ternary code + 16-bit scale amortized over 128 weights); counting the small set of tensors held above the ternary representation brings the model as a whole to **1.72 bits/weight** — an idealized \~9.3x reduction vs FP16.

The weights are stored in a **rotated basis**: each matrix is transformed blockwise by an orthogonal Hadamard rotation before the ternary assignment, and the runtime applies the matching transform to activations. The rotation is folded into the stored weights offline, so it costs no extra bits and no extra weight traffic; the packed model declares its rotation as metadata, so a runtime either applies the matching transform or refuses to load the file.

### Memory Requirement

| Format | True bits/weight | Size | Reduction |

| :------------------------------ | ---------------: | ----------: | ----------: |

| FP16 (baseline) | 16.0 | \~54 GB | 1.0x |

| Ternary g128 (ideal) | 1.72 | 5.8 GB | \~9.3x |

| **GGUF PTQ1_0** (dense trits) | **1.75** | **5.95 GB** | **\~9.0x** |

| **GGUF PQ2_0** (2-bit slots) | **2.13** | **7.21 GB** | **\~7.5x** |

Practical deployment needs packing formats that efficient kernels can consume, and this repo ships two: **PTQ1_0** packs trits densely and lands essentially on the information-theoretic target, while **PQ2_0** stores each trit in a 2-bit slot, trading footprint for cheaper unpacking. Neither is uniformly faster — see the throughput table below for where each wins. These sizes describe the language model alone, the only component that must stay resident for text inference; 26.2M parameters (**0.0976%** of the language model — the recurrent state path of the linear-attention layers, plus the normalization weights) remain in higher precision and are counted in the 1.72 figure.

Unlike conventional low-bit builds — whose advertised labels understate their true average bit-width (a widely-used "2-bit" build of Qwen3.8-27B is really 2.8 bits/weight at 9.4 GB) — the Bonsai representation carries a bit-width that matches its name.

### Shipped Components

The vision tower ships alongside the language model as an optional component (on-disk sizes):

| Component | Pack | Size | Residency |

| :------------- | :--------------------------------- | -------: | :--------------------------------- |

| Language model | ternary g128 (PTQ1_0) | 5.95 GB | resident |

| Language model | ternary g128 (PQ2_0) | 7.21 GB | resident |

| Vision tower | mmproj (Q8_0) | 0.63 GB | optional — multimodal input only |

| Vision tower | mmproj BF16 (reference) | 0.93 GB | optional |

The Q8_0 file carries the vision tower in an 8-bit container. It is usually offloaded, loaded only when an image actually arrives, so text-only serving never pays for it.

## Best Practices

### Generation Parameters

We recommend using the following sets of sampling parameters for generation:

> - Thinking Mode: `temperature=1.0`, `top_p=0.95`, `top_k=20`, `min_p=0.0`, `presence_penalty=0.0`, `repetition_penalty=1.0`

> - Instruct (or non-thinking) mode: `temperature=0.7`, `top_p=0.80`, `top_k=20`, `min_p=0.0`, `presence_penalty=1.5`, `repetition_penalty=1.0`

These match the base model's own `generation_config.json` and are the values carried in the GGUF

metadata (`general.sampling.*`), so a client that reads model defaults will use them without being

told. They are also the settings used for the reported benchmark results (thinking mode).

The model uses `xhigh` reasoning effort by default; use `medium` for shorter responses and a balance of speed and accuracy. `low` reasoning effort is not supported and when selected the model will behave close to `xhigh`.

### System Prompt

You can use a simple system prompt such as:

```

You are a helpful assistant

```

### Choosing a Packing

**PQ2_0** is the faster decode on H100, A100, and the Blackwell cards, is faster at prompt processing everywhere, and is the pack measured on Apple Silicon. **PTQ1_0** is the faster decode on the Ada-generation cards and the L4, and is the pick wherever memory is tightest. See the throughput table below.

## Quickstart

> **[PrismML-Eng/Bonsai-demo](https://github.com/PrismML-Eng/Bonsai-demo) is the source of truth for

> running these models.** It carries the tested setup for every backend, pins a known-good binary

> release, and is kept current as the kernels move. Where anything here disagrees with it, it is right.

### These files need our llama.cpp build

The ternary hybrid-attention kernels live in the

[PrismML-Eng/llama.cpp](https://github.com/PrismML-Eng/llama.cpp) fork. **Stock llama.cpp will not

run these files.** It rejects `PQ2_0` and `PTQ1_0` as unknown types, and it loads `Q2_0` without any

warning and produces garbage, because it has no Hadamard activation runtime. Use a binary from the

fork.

```bash

# prebuilt, pick the archive for your platform

# https://github.com/PrismML-Eng/llama.cpp/releases/latest

tar -xzf llama-<tag>-bin-<platform>.tar.gz -C bin --strip-components=1

# or build it

git clone https://github.com/PrismML-Eng/llama.cpp && cd llama.cpp

cmake -B build -DGGML_CUDA=ON && cmake --build build -j # drop -DGGML_CUDA=ON on macOS, Metal is default

```

```bash

hf download prism-ml/Ternary-Bonsai-2-27B-gguf Ternary-Bonsai-2-27B-PQ2_0.gguf --local-dir .

```

```bash

./bin/llama-cli -m Ternary-Bonsai-2-27B-PQ2_0.gguf \

-ngl 99 -fa on -c 32768 \

--temp 1.0 --top-p 0.95 --top-k 20 \

-p "Explain quantum computing in simple terms." -n 256

```

The binary is `./bin/llama-cli` from an extracted release archive, or `./build/bin/llama-cli` if you

built the fork yourself. `-ngl 99` offloads every layer, `0` is CPU-only; `-c` sets the context, up

to 262144.

This is a reasoning model and it thinks by default. For the server, tool calling, reasoning budgets,

image input with the `mmproj` file, and speculative decoding, follow

[Bonsai-demo](https://github.com/PrismML-Eng/Bonsai-demo), which ships run scripts that pick the

right flags for your hardware.

## Cross-Platform Throughput

`tg128` is token-generation throughput over 128 generated tokens (the memory-bandwidth-bound, interactive phase); `pp512` is prompt-processing throughput over 512 input tokens (the compute-bound phase). Both in tokens/s, measured with `llama-bench` on these GGUF packs (custom low-bit kernels), at batch size 1 and depth 0 with no vision tower. NVIDIA energy is board power including HBM/GDDR.

| Platform | PQ2_0 TG128 | PQ2_0 PP512 | PQ2_0 J/tok | PTQ1_0 TG128 | PTQ1_0 PP512 | PTQ1_0 J/tok |

| :--------------------------- | ----------: | ----------: | ----------: | -----------: | -----------: | -----------: |

| RTX 5090 (32 GB) | **129.9** | 3893 | **1.95** | 120.5 | 1805 | 2.15 |

| RTX PRO 6000 Blackwell | 124.8 | **4020** | 2.49 | 117.9 | 1972 | 2.77 |

| H100 SXM (80 GB) | 113.9 | 2830 | 2.69 | 86.9 | 1237 | 3.18 |

| RTX 6000 Ada (48 GB) | 82.8 | 2431 | 2.51 | **90.4** | 1657 | 2.49 |

| RTX 4090 (24 GB) | 81.2 | 3124 | 2.99 | **91.1** | 1645 | 2.58 |

| L40S (48 GB) | 74.4 | 2868 | 3.24 | **81.8** | 1543 | 2.82 |

Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf