Perplexity Details Its GPU Embedding Stack: How Ivy, Tulip and ROSE Serve pplx-embed
Perplexity details its GPU embedding serving stack (Ivy, Tulip, ROSE), which reuses LLM prefill/decode kernels, CUDA graphs, and LazyTensors to cut launch overhead.
Perplexity engineers published a deep dive on the serving infrastructure behind pplx-embed, used across Perplexity Search and its API platform. The stack comprises Ivy (Rust HTTP gateway), Tulip (gRPC scheduling and batching), and ROSE (Runtime-Optimized Serving Engine), which reuses LLM prefill and decode kernels rather than running a separate embedding engine. Optimizations include whole-model CUDA graphs with lazy capture and a LazyTensor abstraction that overlaps CPU batch preparation with in-flight GPU work. Benchmarks are reported against vLLM v0.22.0 in BF16, with FlashAttention 4 generally fastest but FlashInfer 3 winning on Qwen-based models at very long sequence lengths.
- Ivy, Tulip, and ROSE split gateway, scheduling, and inference responsibilities
- Embedding serving reuses LLM prefill/decode kernels; no separate engine was built
- Whole-model CUDA graphs with lazy capture cut kernel launch overhead
- Latency tracks token count, not sequence count, near ~512-token saturation on sub-1B models
- FlashInfer 3 outperforms FlashAttention 4 on Qwen-based models at long sequence lengths
- Vendors
- Perplexity
- Products
- pplx-embedIvyTulipROSEvLLM
- Organizations
- Perplexity
- AI models
- Qwen
From the original article
4 min read at marktechpost.com
Retrieval quality in an AI search product is bounded by two things: how good the embedding model is, and how cheaply you can run it across an index. This week, Perplexity Engineering team published Fast Embeddings on GPUs, an under-the-hood account of the second — the serving infrastructure behind pplx-embed and the ranking models used […]
By Asif Razzaq · marktechpost.com
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.