A Three-Layer Caching Architecture for Low-Latency LLM Web Search on Commodity CPU Hardware
OreoLook's three-layer Redis caching architecture cuts redundant LLM calls and embedding work for CPU-hosted web-search answer generation.
The paper describes a three-layer caching architecture for OreoLook (formerly lixSearch), an open-source LLM answer engine: a Redis session context window with Huffman-compressed disk overflow, a semantic query cache matching rephrasings via embedding cosine similarity, and a URL embedding cache deduplicating embedding computations. Deployed on a single 8-vCPU Intel Cascade Lake server with 30 Hypercorn workers across three containerized replicas, it achieved an 89.3% aggregate Redis keyspace hit rate, 0.1 ms read latency, and 1.38 MB memory overhead. An LRU eviction daemon migrates idle sessions to disk and rehydrates them for resumption hours or days later.
- Semantic query cache catches rephrased queries via embedding cosine similarity to skip redundant LLM calls.
- Deployed on one 8-vCPU Cascade Lake server with 89.3% aggregate cache hit rate and 0.1 ms reads.
- LRU daemon migrates idle sessions to compressed disk archives and rehydrates them on demand.
- Vendors
- Intel
From the original article
1 min read at huggingface.co
AI-powered search products such as ChatGPT search, Google's AI Overviews, and Perplexity provide LLM-synthesized answers grounded in live web results. We developed OreoLook (formerly lixSearch), an open-source answer engine using automated browser agents and provider-routed LLM inference.
By Ayushman Bhattacharya, Nihal Gazi · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.