FlashVector: Agent for Hierarchical Model Serving Stack Optimization
FlashVector agent optimizes all layers of Unity's ad-serving stack, delivering up to 2x model-server throughput and 1.98x latency speedup in production.
FlashVector is an agentic system that optimizes performance across GPU kernels, ML framework computation graphs, model servers, and on-demand feature processing. Deployed in Unity's Vector advertising platform, it achieved up to 2x model-server throughput increase, 1.98x latency speedup, and 1.6x feature-store throughput gain. Optimizations spanned NVIDIA Triton's C++ codebase and the Python feature transformation service, demonstrating extensibility beyond single-kernel tuning.
- Generalizes single-kernel agent paradigm to heterogeneous serving stacks
- Production deployment at Unity's Vector advertising platform
- Up to 2x throughput and 1.98x latency speedup on model server
- Optimizations span Triton C++ and Python feature services
From the original article
1 min read at arxiv.orgModel serving is one of the largest cost drivers in production recommender systems. Maximizing its throughput requires navigating a deeply layered hierarchy: GPU kernels, the ML framework computation graph, the model server, and on-demand feature processing -- each demanding specialized domain expertise.
By Qi Wu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.