AutoCompress Framework for Hardware-Guided Model Compression
Original title: AutoCompress: Automated Hardware-Guided Model Compression
AutoCompress optimizes LLM architectures for latency and memory constraints, improving accuracy over Minitron by up to 6.57 percentage points.
The paper presents AutoCompress, a hardware-guided compression framework that uses differentiable architecture search to meet serving-engine constraints on latency, throughput and memory. It produces heterogeneous models validated on end-to-end measurements. AutoCompress compresses Nemotron 3.5 Super from 120B to 74B parameters for B200 serving at 128K context while retaining 95% of mean benchmark score. It improves average accuracy by 6.57, 2.35 and 0.57 percentage points on RTX 5070, RTX 5080 and H100.
- Jointly optimizes global and per-layer structure
- Improves accuracy over Minitron by 6.57 pp on RTX 5070
- Compresses Nemotron 3.5 Super to 74B parameters
- Validates on batched serving and long-context coding
- Retains 95% mean benchmark score for B200
- Organizations
- Hugging Face
- AI models
- Nemotron 3.5 Super
From the original article
1 min read at huggingface.co
LLM pruning must preserve quality while meeting hardware- and workload-specific constraints on latency, throughput, and memory. Existing pruning workflows often target parameter counts, requiring practitioners to manually select model sizes that meet latency, throughput, and GPU memory limits.
By Sheldon Liang, Ali Taghibakhshi, Saurav Muralidharan · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.