JustFit: 200K-Token LLM Serving on a 24 GiB Laptop with Just-in-Time State Management
JustFit MLX runtime serves 200K-token contexts for Qwen3.8-27B on a 24 GiB MacBook via just-in-time state management.
JustFit is an MLX-based inference runtime combining KVExec for compressed KV execution, PhaseSwap for component residency, and StateTrans for state-preserving serving transitions, independent of weight quantization. On a 24 GiB M4 Pro MacBook running Qwen3.8-27B MXFP4, it completed 196,608 input and 16,384 output tokens, raising single-request context from the mlx-vlm baseline's 30,720 positions to 212,992 (6.93x). Performance tests show 19.11 tokens/s on a 32K-input probe with a 16,374 MiB median peak footprint, and the runtime answered 29 of 30 AIME 2026 problems correctly.
38