How Language Models Track Word Order, and Why Newer Designs Are Dropping Some of It

Language models track word order by attaching position information to tokens or to attention. Early GPT-2 used learned position vectors, the original transformer used fixed sine and cosine functions, and RoPE rotates query and key vectors by token position. Newer designs use partial RoPE or skip RoPE in some layers, while RoPE remains the most common method.

Word order matters to meaning. “The red fox jumps over the gray fox” and “The gray fox jumps over the red fox” contain the same words, but they describe different animals doing the jumping. A standard attention mechanism, which lets each token weigh every other token, would treat the two sentences the same way unless the model is told where each word sits. A new technical explainer lays out how large language models solve that problem, and how the methods are changing in 2026.

How do language models know where each word sits?

The guide starts with the simplest approach. GPT-2 learns a table of position vectors during training, one for each sequence position, and adds the matching vector to each token embedding before the first transformer block. The original GPT-2 model has 1,024 learned position vectors, which means it can only handle inputs of 1,024 tokens or fewer. Extending the context means extending that table, and the table must be sized before training begins.

The cost of that table grows quickly. According to the walkthrough, position embeddings make up 0.30 percent of the total weights in GPT-2 Medium at 1,024 positions. At a context length in the 100k range, the author estimates they would account for more than 22 percent of the model's weights, which would be uneconomic.

The 2017 transformer paper, “Attention Is All You Need,” used a different route. Its position vectors come from fixed sine and cosine functions, which do not change during training. Those functions can be evaluated beyond the training positions, though the guide notes that this does not guarantee good performance there. The authors also hypothesized that the sine-and-cosine design would help attention learn relationships such as “three tokens earlier,” because a fixed offset produces the same rotation no matter where it falls in the sequence.

What is RoPE and how does it encode position?

RoPE takes that idea further by moving the position signal out of the input and into attention itself. Introduced by Su and colleagues in the 2021 RoFormer paper, the method rotates each query and key vector based on its token position before the two are multiplied. Value vectors skip the rotation. Because the angle between a query and a key depends only on how far apart they are, the attention score reflects relative distance. The guide shows that shifting both tokens by the same amount leaves their dot product unchanged.

RoPE has become a staple of open-weight models since the original LLaMA. Proprietary models may use it too, the guide notes, but outside details are hard to confirm.

Recent designs are bending that pattern. Partial RoPE rotates only a subset of each head's features. MiniMax-M3, for example, uses 64 rotary dimensions within each 128-dimensional head. The main payoff is memory: the precomputed cosine and sine tables shrink by half. For MiniMax-M3's 1-million-token context, the guide estimates the savings at 0.134 GB if the tables are stored in BF16. The KV cache still stores full key and value vectors, so the saving applies only to the tables.

Other models skip RoPE in some layers altogether. SmolLM3, a 3-billion-parameter model, applies RoPE in three layers, skips it in the fourth, and repeats that pattern across 36 layers. The SmolLM3 authors said the layer pattern improved long-context performance. Skipping layers that use no explicit positional encoding, known as NoPE, is possible because the causal mask already gives attention some sense of order. A query can only see itself and earlier tokens, so the averaging behavior of attention carries position information forward.

A third approach, found in Thinky's Inkling, adds a learned relative position bias directly to attention scores before the softmax. The bias depends on the query token's hidden state and how many positions back the key sits. The guide points out that T5, Google's 2019 model, used a simpler form of the same idea.

Are newer language models still using RoPE?

To see which methods are used where, the author grouped entries from an LLM architecture gallery by release date and compared 2025 releases with those from 2026. RoPE in every softmax attention layer is still the most common choice. Its share fell from 2025 to 2026, and it fell again in the latter half of 2026. Mixed RoPE and NoPE designs are growing, though the groups are still small. Among models that use RoPE, partial RoPE went from a minority in 2025 to a majority in both 2026 groups.

The guide's conclusion is that RoPE remains the dominant position method, and the author expects it to carry into 2027. The author also suggests that more Inkling-style designs could appear next year. The walkthrough is aimed at readers with some background in transformer basics, and it is the first in a planned set of fundamentals explainers.

This article was written with the assistance of AI.
News Factory APP - agentic news to boost your SEO & AEO.