Token-Energy Law ties LLM decode joules to active bytes and time
Original title: The Token-Energy Law of LLM Decode: Active Bytes and Realized Time Account for Joules per Token
Held-out energy estimates on Hopper and Blackwell stayed within 3.73% of measured means.
A paper presents the Token-Energy Law, an affine model of GPU-board energy per output token for dense, full-history decoding. It uses active bytes—shared dense weights versus private KV history—and realized decode time, with coefficients fit per model and hardware setup. Experiments cover three models in BF16 on NVIDIA Hopper and Blackwell, including two-GPU tensor parallelism on Hopper.
- Batching amortizes shared-weight work; longer histories raise private KV demand.
- Tests include two-GPU tensor parallelism on Hopper.
- Fixed coefficients plus observed decode time produced the held-out estimates.
- The law targets dense full-history decoding, not other serving modes.
From the original article
1 min read at arxiv.orgEnergy-efficient deployment of large language models requires understanding how model architecture, hardware, and serving settings shape energy per generated token. We introduce the Token-Energy Law, a compact, interpretable model of GPU-board energy per output token for dense, full-history decoding. It relates token energy to two core dimensions: active bytes and realized decode time.
By Chenxu Niu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.