Flide accelerates decision inference with exact state forking
Original title: Flide: Accelerating Decision Inference with Exact State Forking
Flide decouples logical tree structure from execution in a flattened token stream using ForkAttention, achieving up to 9.5x speedup over vLLM while preserving exact probabilities.
Decision models like Jev provide rapid categorical reasoning for LLM agents but suffer from prefix recomputation overheads in existing engines. Flide uses ForkAttention to navigate request dependencies across attention and Gated DeltaNet layers via GPU-resident indices. Evaluations on hybrid and decision models confirm exact arithmetic equivalence to isolated forward passes.
- Decouples logical tree from physical execution in single token stream
- GPU-resident metadata enables diverse request mixtures
- Answers queries up to 9.5 times faster than vLLM and SGLang
- Stays closest to float32 reference in total variation distance
From the original article
1 min read at arxiv.orgDecision models such as Jev have emerged as a highly promising paradigm for LLM agent systems, providing lightweight and reliable categorical reasoning for rapid System One execution. However, existing generation-oriented inference engines fail to support these workloads effectively, suffering from severe prefix recomputation overheads or distorted output probabilities.
By Bobo Li · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.