PLatour Enables Prefill-to-Autoregressive Cross-Model Transfer
PLatour framework shows promise in balancing capabilities and costs but gains may not require true sender-receiver communication.
Stories per day · 30 days
12 stories, newest first · one card per story
Stories tagged #inference-optimization
On-Demand Attention: Language Models Know When to Recall
On-Demand Attention uses a lightweight recall head to selectively invoke global attention, cutting long-context decoding costs in vLLM with minimal quality loss.
To Copy or Not to Copy: Controlling Speculative Decoding via Intrinsic Model Signals
SwitchSD uses internal LLM signals to adaptively switch between neural drafting and context-based copying in speculative decoding, boosting throughput up to 15%.
DeepSeek-v4.1 Flash: Pushing the Limits of KV Cache Compression
DeepSeek-V4.1 Flash is a 552B-parameter multimodal MoE model with 1M-token context achieving 4x KV cache compression for long-horizon agent workloads.
11
Nunchux AI Introduces VC-Attention: A Training-Free Low-Bit Attention Kernel That Speeds Up Video Diffusion Transformers
Nunchux AI introduces VC-Attention, a training-free low-bit attention kernel that speeds up video diffusion transformers up to 3.58x.
6
rMuscle: Robotic Muscle Memory for Efficient Vision-Language-Action Model Inference
rMuscle, a caching-based inference framework for vision-language-action models, achieves 1.29-1.42x speedups on RTX 4090 and Jetson Thor while preserving success rates.
Fathom: Per-Query Read Depth for Sparse Decoding over Offloaded KV Caches
Fathom introduces per-query bit-depth scanning of offloaded 4-bit KV caches, making long-context decoding 1.67x faster on Qwen3-8B at one million tokens.
VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention
VC-Attention is a training-free low-bit attention method for diffusion transformers, achieving 1.46-1.59x kernel speedups on datacenter GPUs with higher fidelity.
ukisai/Swift-Qwen3.8-27b — new model trending #30 on Hugging Face
UkisAI releases Swift-Qwen3.8-27B, a Qwen3.8-27B derivative using 58.3% fewer thinking tokens with <1% performance loss and ~1.95x speed-up.
Online Draft Co-Training for Speculative Decoding in Large-Scale, Long-Context RL Post-Training
NVIDIA researchers detail an end-to-end system for online draft co-training that speeds speculative decoding in large-scale long-context RL post-training.
BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference
BeaconKV introduces training-free KV cache compression using beacon queries, cutting long-reasoning inference memory up to 5.8x while preserving accuracy.
DataGrout helps enterprises control AI usage, governance and LLM costs
SelectHub launched DataGrout, an LLM cost and governance platform combining dynamic context pruning, symbolic inference and MCP gating, claiming about 60% token reduction.
4