知Yizhi Daily
← Back to this edition

AI

Technical Long-Read Explains Positional Encoding Mechanisms in Modern Large Language Models

Edition 2026-11-16 · 08:00 Beijing time

A technical tutorial discussed on Hacker News systematically explains how large language models encode and process token positions, covering classic absolute position embeddings, relative position methods, the RoPE mechanism, and positional encoding choices in open-weight models in 2026. The article includes visualizations, a small amount of mathematical derivation, and PyTorch implementations, aimed at readers who already understand Transformer basics.

The article aims to explain how large language models obtain and process token position information. The author says that as large model architectures continue to evolve, while preparing an article on 2026 architecture trends he found that position-information methods had not previously been specifically introduced, so he organized this part into a standalone foundational tutorial. Its content includes different positional encoding approaches, how RoPE works, and schemes used by recent open large models.

The tutorial begins with the basic flow of large language models: a tokenizer first converts input text into token IDs, the input embedding layer then maps these IDs into numerical vectors, Transformer blocks subsequently modify the vectors through attention and feed-forward layers, and the output layer produces logit scores for the next token. The article uses GPT-2 and the newer MiniMax-M3 as architecture examples to show where position information sits in the overall path.

The article says it will explain classic and contemporary methods through visualized examples, mathematical explanations, and a small PyTorch implementation, including learned absolute position embeddings, relative position embeddings, and RoPE, and will further discuss how modern architectures adjust or omit explicit positional encoding. The author also previews position embedding choices in open-weight models in 2026.

The article assumes readers are familiar with the basic architecture of large language models, especially query, key, and value vectors in self-attention. The author notes that readers who already understand position embeddings can jump directly to the chapter on position choices in modern architectures; the article is better read in a browser so the table of contents can be used for navigation.

Original sources

Hacker News ↗
Back to this editionOpen interactive reader