DuoLM reuses compact item embeddings for faster LLM recommendations
Original title: DuoLM: Fast and Effective Personalized Recommendation with Language Models
The paper reports about 26,400 candidate scores per second on one H100, roughly 60 times a vLLM-served teacher.
DuoLM combines an encoder LLM that stores compact item representations offline with a personalized ranker that uses user history and feedback. Both models are trained end to end with recommendation supervision and teacher distillation. The serving runtime uses virtual token IDs, shared history computation, and CUDA Graph replay. The authors report quality comparable to the expert teacher and much higher throughput on a single H100.
- Encoder embeddings are computed offline and reused at serving time.
- Training uses recommendation supervision plus teacher distillation.
- Runtime tricks include virtual token IDs and CUDA Graph replay.
- Reported throughput is about 26,400 scores per second on one H100.
From the original article
1 min read at huggingface.co
Language models can make personalized recommendations from rich item content, user history, and feedback. However, evaluating many candidates with full-text inputs incurs substantial online computation and limits serving throughput. We introduce DuoLM, a framework for fast and effective personalized recommendation that integrates a compact recommendation model with an efficient serving runtime.
By Jiang Wu, Yutong Song, Hongbo Wang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.