JustFit: 200K-Token LLM Serving on a 24 GiB Laptop with Just-in-Time State Management
JustFit MLX runtime serves 200K-token contexts for Qwen3.8-27B on a 24 GiB MacBook via just-in-time state management.
JustFit is an MLX-based inference runtime combining KVExec for compressed KV execution, PhaseSwap for component residency, and StateTrans for state-preserving serving transitions, independent of weight quantization. On a 24 GiB M4 Pro MacBook running Qwen3.8-27B MXFP4, it completed 196,608 input and 16,384 output tokens, raising single-request context from the mlx-vlm baseline's 30,720 positions to 212,992 (6.93x). Performance tests show 19.11 tokens/s on a 32K-input probe with a 16,374 MiB median peak footprint, and the runtime answered 29 of 30 AIME 2026 problems correctly.
- KVExec, PhaseSwap, and StateTrans coordinate just-in-time state materialization and release
- 212,992 single-request context positions on 24 GiB, 6.93x the mlx-vlm baseline
- 19.11 tokens/s on 32K-input probe with 16.4 GiB median peak footprint
- Independent of weight quantization; solves 29 of 30 AIME 2026 problems
Full article154 words · extracted from arxiv.org · click to collapse
Capable open-weight models make local coding and reasoning attractive, but their context and execution state strain laptop memory. We present JustFit, an MLX-based inference runtime that combines KVExec for compressed KV execution, PhaseSwap for component residency, and StateTrans for state-preserving serving transitions. These mechanisms fuse reconstruction and coordinate just-in-time materialization and release, independently of model-weight quantization. In full-execution capacity tests on a 24 GiB M4 Pro MacBook running Qwen3.8-27B MXFP4, three independent runs complete 196,608 input and 16,384 output tokens, increasing completed single-request context from the mlx-vlm baseline's 30,720 positions to 212,992 (6.93x); a separate two-request run retains 229,376 positions in aggregate. In separate performance tests, a 32K-input, 64-output probe reaches 19.11 tokens/s, and a repeated 32K+6K workload has a median peak process footprint of 16,374 MiB. The integrated runtime answers 29 of 30 AIME 2026 problems correctly, showing how compact state and lifetime-aware execution expand local serving capacity while supporting extended generated reasoning.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.17475