Skip to content
NewA calmer, faster ZeroHour

Understanding positional embeddings in modern LLMs

Original title: Understanding Positional Embeddings in Modern LLMs

Hacker News · AIby vismit2000 ()14 min read1
AI summary · x-ai/grok-4.7Summaries by Vani AI

The walkthrough covers absolute embeddings, RoPE, and a small PyTorch implementation, then surveys 2026 open-weight choices.

Sebastian Raschka published a technical explainer on how LLMs encode token position. It contrasts GPT-2, which adds a position vector to token embeddings, with MiniMax-M3, which applies RoPE inside attention in each transformer block. The piece is framed as a building block for a later article on 2026 LLM architecture trends and includes visual examples and a small PyTorch implementation.

Key facts
  • GPT-2 adds a learned position vector before the first transformer block.
  • MiniMax-M3 injects position with RoPE inside attention rather than at the embedding layer.
  • A causal mask supplies some implicit order even without explicit positional encodings.
  • Section 6 surveys positional encoding choices in 2026 open-weight models.
Explain this AI news with Vani AIWhat was announced, what is new and why it matters, in a new chat

From the original article

14 min read at magazine.sebastianraschka.com

This article explains how LLMs encode and process token position information by covering classic and contemporary methods like RoPE. It’s essentially a comprehensive journey where we will work through the mechanisms with visual examples, a little math, and a small PyTorch implementation. And, of course, we will then also look at how modern architectures adapt or omit explicit positional encodings.

Hacker News · AI

By vismit2000 · magazine.sebastianraschka.com

Read the full article at Hacker News · AI

A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.