Positional Encoding in Transformers, Explained

Positional encoding is a set of values added to a token’s embedding that tells a transformer where that token sits in the sequence. Self-attention on its own has no built-in sense of order — it looks at every token relative to every other token in parallel, so without positional information “the dog bit the man” and “the man bit the dog” would produce identical attention patterns. Positional encoding is what lets the model tell them apart.

Why attention needs help with order

Self-attention computes relevance scores between every pair of tokens using their query and key vectors, then blends value vectors accordingly. That mechanism is inherently permutation-invariant: shuffle the input tokens, and the set of pairwise scores shuffles right along with them, but the underlying computation doesn’t change. This is actually one of attention’s strengths — it’s what lets a transformer process an entire sequence in parallel rather than one token at a time, unlike the recurrent networks transformers replaced. But it means order has to be injected some other way, since the architecture itself doesn’t encode it.

The fix is simple in concept: add a vector that encodes “this is token 3” (or “this is 3 tokens from the start”) directly into each token’s embedding before it enters the attention layers. The model then has access to positional information as part of the same representation it already uses for meaning.

Sinusoidal encoding, the original approach

The original transformer paper used a fixed, non-learned scheme: each position gets a vector built from sine and cosine functions at different frequencies, one pair per dimension of the embedding. Low dimensions oscillate quickly across positions; high dimensions oscillate slowly. The result is a unique fingerprint for every position, and — critically — a mathematical property where the encoding for position p + k can be expressed as a fixed linear function of the encoding for position p. That made it easier, in principle, for the model to learn to attend by relative offset rather than only by absolute position.

Sinusoidal encodings are deterministic: they aren’t learned during training and can be computed for any position, including ones longer than anything seen in training data. In practice, though, models using this scheme still tend to degrade once inference sequences run well past the lengths they trained on.

Learned absolute encodings

An alternative is to just learn a positional embedding table the same way the model learns token embeddings — one vector per position, trained by gradient descent alongside everything else. This is simpler to implement and, on the sequence lengths a model actually trains on, often performs comparably to sinusoidal encoding.

The downside is inflexibility: a learned table has a fixed number of rows, one per position up to some maximum. Feed the model a sequence longer than that maximum and there’s no encoding to look up — the context window is hard-capped by however many position embeddings were trained.

Relative and rotary position encodings

Modern large language models mostly use approaches that encode relative position — the distance between two tokens — rather than each token’s absolute index. Rotary Position Embedding (RoPE), used in most current open-weight and proprietary LLMs, rotates each query and key vector by an angle proportional to its position before the attention score is computed. The dot product between a rotated query and a rotated key ends up depending only on the difference between their positions, not their absolute values, so the same relative-distance pattern generalizes more naturally to positions the model never explicitly trained on.

This matters directly for how far a model’s context can stretch. Techniques for extending context length after training — interpolating or rescaling RoPE’s rotation frequencies — are a big part of how vendors take a model trained at, say, 8,000 tokens and get it working reasonably well at ten times that length. See our explainer on RAG versus long-context LLMs for how that context-length ceiling shapes real application design.

Absolute vs relative: a comparison

	Sinusoidal (absolute)	Learned (absolute)	RoPE (relative)
Learned or fixed	Fixed formula	Learned during training	Fixed formula, applied to Q/K
Extrapolation beyond training length	Weak in practice	None past max trained position	Better, with scaling tricks
Captures relative distance directly	Indirectly	No	Yes, by construction
Used in	Original 2017 transformer	Early encoder models like BERT	Most modern LLMs

Why this is easy to overlook

Positional encoding is a small piece of a transformer’s forward pass — a handful of lines compared to the attention and feed-forward blocks that do the heavy lifting — but it has an outsized effect on a model’s practical limits. The choice of scheme is a major factor in how gracefully a model handles sequences longer than what it saw during training, which in turn shapes decisions about chunking documents for retrieval-augmented generation versus just stuffing more text into a single prompt. It also interacts with the KV cache: because RoPE rotates queries and keys based on position, cached keys computed at one position remain valid as generation proceeds, which is part of why it composes cleanly with efficient inference.

The takeaway

Self-attention has no native concept of sequence order, so transformers add positional encoding to token embeddings to supply it. Early models used fixed sinusoidal signals or learned position tables, both tied to absolute position; most current LLMs use rotary encodings that bake relative distance directly into the attention computation, which generalizes better to sequences longer than anything seen in training. It’s a small architectural detail with a large effect on how far a model’s context window can realistically stretch.