Flide answers decision queries up to 9.5 times faster than vLLM
Original title: Flide: Accelerating Decision Inference with Exact State Forking
ForkAttention keeps each query branch identical to an isolated forward pass while reusing GPU-resident graphs and checkpoints.
Generation-oriented inference engines recompute prefixes or distort probabilities for categorical decision workloads such as Jev. Flide separates a request's logical tree from one flattened token stream and uses ForkAttention to follow GPU-resident indices across standard attention and Gated DeltaNet layers. The authors report closer agreement with a float32 reference than vLLM and SGLang, plus support for mixed request graphs and resumed agent histories.
Unconfirmed: Speed and exactness results are author-reported and not independently confirmed here.
- Flide reports answers up to 9.5 times faster than vLLM and SGLang.
- It claims exact-arithmetic agreement with an isolated forward pass.
- Total variation distance stayed closest to a float32 reference.
- Checkpoints from earlier passes let expanding agent histories resume.
- AI models
- Jev
From the original article
1 min read at huggingface.co
Decision models such as Jev have emerged as a highly promising paradigm for LLM agent systems, providing lightweight and reliable categorical reasoning for rapid System One execution. However, existing generation-oriented inference engines fail to support these workloads effectively, suffering from severe prefix recomputation overheads or distorted output probabilities.
By Bobo Li, Jiajia Song, Hao Fei · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.