Paragraph Boundaries Are Not White Space:Compression Depth as the Signature of Hierarchical Structure
Compression depth, not location, distinguishes real paragraph structure under hierarchical rotary positional encoding.
Researchers use hierarchical rotary positional encoding (hRoPE), representing paragraph, sentence, and token indices as separate channels. Holding the token sequence fixed and intervening on the paragraph coordinate, they find cross-paragraph attention is compressed versus a token-distance-matched baseline in every corpus. Density-matched random labels compress more shallowly, so compression depth—not mere compression—marks genuine paragraph structure. Among eight corpus quantities, embedding-based coherence comes closest but does not fully reproduce the cross-corpus ordering of depth.
- hRoPE encodes paragraph, sentence, and token indices as separate channels.
- Real paragraph labels compress cross-paragraph attention more deeply than random labels.
- Compression depth, not location, is the reproducible structural signature.
- Embedding-based coherence comes closest but does not fully match depth ordering.
From the original article
1 min read at huggingface.co
Standard positional encodings represent position as a one-dimensional reading-order coordinate, but reading order alone does not determine hierarchical textual structure.
By Shuyang Xiang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.