Can gzip be a language model?
A developer shows gzip can continue text by beam-searching byte sequences that compress best against a primed corpus.
A developer demonstrates language modeling with gzip by scoring candidate byte continuations using DEFLATE compressed length. A corpus is placed in zlib's 32 KiB window, and beam search keeps the spans that compress best before committing them. Single-byte greedy selection fails because compressed length is an integer and often unchanged. Text primed on tiny Shakespeare is only partly coherent, and the author notes earlier compression-only generation performed poorly without beam search.
- DEFLATE scores a continuation by the compressed length of context plus candidate.
- Beam search over corpus bytes avoids ties from integer-length quantization.
- Only a recent output tail stays in context to limit copy loops.
- The demo is one standard-library Python file using zlib.
From the original article
4 min read at nathan.rsA while back I wrote about language modeling without neural networks , where I generated Shakespeare with an unbounded n-gram model: no weights, no training, just counting. Fortuitously, I came across the paper Language Modeling is Compression , which mentioned the compression–prediction equivalence : every prediction model is inherently a compressor, and all compression algorithms are prediction models .
By networked · nathan.rs
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.