Replacing Large Language Models with Jev Decision Models for Low-Latency Edge Service Orchestration
Jev decision models cut median edge-orchestration latency 23-65% versus the fastest LLM on 8,280 requests.
The paper integrates Jev's decision API into edge service orchestration, extracting four to eight bounded intent fields and applying a shared validator, admission policy, and scheduler. Across 33 conditions and 8,280 verified requests, Jev cuts median decision latency by 22.7-64.5% versus the fastest compared LLM, with latency largely insensitive to input size, contract width, or catalog size. On four-field contracts, per-correct-decision API fees fall 59.7-80.9% at a cost of a few exact-match points, while wider contracts limit substitution. On a live admission path, Jev keeps 0.91-0.95 of requests exact and on time where LLMs fall below 0.1.
- Jev extracts four to eight bounded intent fields before scheduling.
- Median decision latency falls 22.7-64.5% versus the fastest LLM.
- Four-field API fees drop 59.7-80.9% with a small exact-match loss.
- On the live path, Jev keeps 0.91-0.95 of requests exact and on time.
From the original article
1 min read at huggingface.co
Natural-language service requests can require a language-model decision before execution starts, consuming part of the request's latency budget. We integrate Jev's decision-oriented application programming interface (API) into edge service orchestration to reduce this overhead while retaining service completion.
By Delong Li, Xu Wang, Haochen Gong · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.