JustFit: 200K-Token LLM Serving on a 24 GiB Laptop with Just-in-Time State Management
JustFit MLX runtime serves 200K-token contexts for Qwen3.8-27B on a 24 GiB MacBook via just-in-time state management.
JustFit is an MLX-based inference runtime combining KVExec for compressed KV execution, PhaseSwap for component residency, and StateTrans for state-preserving serving transitions, independent of weight quantization. On a 24 GiB M4 Pro MacBook running Qwen3.8-27B MXFP4, it completed 196,608 input and 16,384 output tokens, raising single-request context from the mlx-vlm baseline's 30,720 positions to 212,992 (6.93x). Performance tests show 19.11 tokens/s on a 32K-input probe with a 16,374 MiB median peak footprint, and the runtime answered 29 of 30 AIME 2026 problems correctly.
- KVExec, PhaseSwap, and StateTrans coordinate just-in-time state materialization and release
- 212,992 single-request context positions on 24 GiB, 6.93x the mlx-vlm baseline
- 19.11 tokens/s on 32K-input probe with 16.4 GiB median peak footprint
- Independent of weight quantization; solves 29 of 30 AIME 2026 problems
From the original article
1 min read at arxiv.orgCapable open-weight models make local coding and reasoning attractive, but their context and execution state strain laptop memory. We present JustFit, an MLX-based inference runtime that combines KVExec for compressed KV execution, PhaseSwap for component residency, and StateTrans for state-preserving serving transitions.
By Yuhua Chen · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.