Positional Encoding

Chapter 1: Sinusoidal Encoding

Vaswani et al. needed a position signal in 2017 and they had one hard constraint: it had to work for sequence lengths the model had never seen. So they built one out of sine waves.

Start with the obvious idea and watch it fail. Add the raw index to every token vector: position 0 adds 0, position 512 adds 512. Now the position term is a hundred times larger than the embedding values, which live around plus or minus one. The token identity gets buried under a number that only says "you are late in the sentence." Bad.

Second try: divide by the max length so positions land in [0, 1]. Now the signal is small enough, but the meaning of "0.5" depends on the total length. Position 5 of 10 and position 500 of 1000 both encode as 0.5. The model cannot tell a short sentence from a long one, and it has no fixed notion of "three tokens back."

Building the Encoding From Scratch

Here is the trick. Think about how binary counting works. The ones bit flips every step, the twos bit every two steps, the fours bit every four. Each bit is a square wave at half the frequency of the one before, and together the bits uniquely identify any integer.

Sinusoidal encoding is that idea in continuous form. Instead of square waves at frequencies 1, 1/2, 1/4, we use sine waves at frequencies that shrink geometrically. Dimension pair 0 oscillates fast, dimension pair 1 slower, and so on down to a wave so slow it barely completes one cycle across the whole context.

Read that as: pos is the position, d is the model width, and i indexes the pair of dimensions. Each pair (2i, 2i+1) gets one angular frequency θ_i = 1 / 10000^2i/d, and we store both the sine and the cosine of pos·θ_i. Storing both is what makes the pair behave like a point on a circle rather than a single wobbling number. Hold that thought: a point on a circle is about to become the entire story.

Hand Calculation: d = 4, Positions 0 to 3

With d = 4 there are two pairs, so two frequencies. Compute them first.

Now evaluate at four positions. Angles are in radians.

Look at what the two pairs are doing. The fast pair (columns 2 and 3) has already swung most of the way around the circle by position 3. It discriminates neighbours sharply. The slow pair (columns 4 and 5) has barely moved: 0.030 radians is under two degrees. It cannot tell position 2 from position 3, but it will still be climbing steadily at position 900, long after the fast pair has wrapped around a hundred and forty times.

Why 10000 as the Base?

The base sets the ratio between the fastest and the slowest wave. With base B, the slowest frequency is 1/B, so its wavelength is 2πB. With B = 10000 the slowest wave has a wavelength of about 62,832 tokens, comfortably longer than any 2017 context window. The fastest wave has a wavelength of 2π ≈ 6.3 tokens, fine enough to separate immediate neighbours.

Pick the base too small and the slow waves wrap around inside your context, creating aliasing: two far apart positions get the same fingerprint. Pick it too large and the slow dimensions are almost constant, wasting capacity. Ten thousand was chosen for a 512-token context. When Llama 3 went to 128k tokens it raised the RoPE base to 500,000 for exactly this reason, which we return to in Chapter 10.

The Property That Almost Gives Relativity

There is one more thing hiding in the sine and cosine pairing, and it is the germ of everything in Chapter 3. Take a single pair at frequency θ and write it as a two dimensional vector:

Now shift by k and expand with the angle addition formulas:

Stack those two lines and you have a matrix times p(pos), and the matrix depends only on k, not on pos. It is a rotation by kθ.

This is why the 2017 paper argued that relative positions are recoverable: the offset shows up as a fixed linear map. But recoverable is not the same as recovered. Nothing in the architecture forces W_Q and W_K to exploit that structure, and the position vector was added to the token vector, so content and position are tangled before attention even sees them. RoPE's whole contribution is to apply the rotation directly to the query and the key, so relativity is not a property the model may discover but a property the arithmetic guarantees.

See It: The Frequency Ladder

Two views of the same object. Heatmap shows the encoding matrix: rows are positions, columns are dimensions, colour is the value. The stripes get wider from left to right, which is the frequency slowing down. Waves plots each dimension pair as its own curve so you can watch the ladder of periods directly. Drag the base slider down to 100 and watch the slow waves start wrapping inside the window.

From Scratch: Build the Table

python
import numpy as np

def sinusoidal_pe(max_len, d_model, base=10000.0):
    pos = np.arange(max_len)[:, None]              # (L, 1)
    i   = np.arange(0, d_model, 2)[None, :]         # (1, d/2)
    theta = 1.0 / (base ** (i / d_model))          # (1, d/2)
    ang = pos * theta                                # (L, d/2)
    pe = np.zeros((max_len, d_model))
    pe[:, 0::2] = np.sin(ang)
    pe[:, 1::2] = np.cos(ang)
    return pe                                     # (L, d)

pe = sinusoidal_pe(4, 4)
print(pe.round(3))
# [[ 0.     1.     0.     1.   ]
#  [ 0.841  0.54   0.01   1.   ]
#  [ 0.909 -0.416  0.02   1.   ]
#  [ 0.141 -0.99   0.03   1.   ]]   matches the hand table

Shapes matter. pe is (L, d) and gets broadcast-added to the token embedding tensor of shape (B, L, d) once, before the first attention layer. It is a buffer, not a parameter: no gradient flows into it, and it costs nothing beyond L·d floats you can also recompute on the fly.

Chapter 2: Learned Tables and Relative Bias

BERT and GPT-2 looked at the sine wave construction and asked the obvious question: why hand-design anything? Make a lookup table with one trainable vector per position, initialise it randomly, and let gradient descent decide what "position 7" should mean.

How the Learned Table Works

You allocate a matrix E_pos of shape (max_len, d). Token at index 5 gets row 5 added to its token embedding. That is the entire mechanism. GPT-2 used max_len = 1024 with d = 768, which is 786,432 extra parameters, about 0.6 percent of the model.

Hand Calculation: Token Plus Position

"cat" at position 0 becomes [0.60, −0.20, 0.20, 0.30]. The same "cat" at position 3 becomes [0.10, 0.40, 0.50, −0.20]. Two genuinely different vectors for the same word, which is exactly what we asked for.

Now compute the attention score between "cat at 0" and "cat at 3" using identity projections. Dot product: 0.60·0.10 + (−0.20)·0.40 + 0.20·0.50 + 0.30·(−0.20) = 0.06 − 0.08 + 0.10 − 0.06 = 0.02. Position changed the score. Fine. But notice what we cannot say: nothing in that number is a function of the offset 3 by itself. It is a function of the pair (0, 3).

The Length Limit

The table has max_len rows. Feed the model position 1024 when it trained on 1024 rows and there is no row to look up. You either crash or you index into uninitialised memory. There is no principled extension, because the model learned each row independently and never learned a rule that generates rows.

Worse, the late rows are undertrained. If most of your training documents are 200 tokens long, rows 800 through 1023 got a fraction of the gradient signal that row 5 got. Quality sags well before the hard limit.

What Relative Encoding Looks Like

Shaw et al. (2018) proposed the fix that names the era. Do not tell the model where each token is. Tell it, for each pair, how far apart they are. Concretely, add a learned scalar to the attention logit that depends only on the offset i − j:

The vector b holds one learned number per offset bucket. T5 refined this into bucketed relative bias: offsets 0 through 7 get their own bucket each, then buckets widen logarithmically out to a maximum distance, and everything beyond lands in the last bucket. Thirty-two buckets cover any distance you like, with fine resolution nearby and coarse resolution far away, which matches how language actually works.

Hand Calculation: Same Distance, Different Encodings

Take two token pairs with the same gap of 2: the pair (3, 1) and the pair (503, 501).

The relative row is the one to stare at. One learned parameter, shared by every pair at distance 2 anywhere in the sequence. Every training example at that offset improves it. That is the capacity argument, and it is why relative schemes generalise to longer sequences with far less pain.

See It: Table versus Band

Left: the learned absolute table, one independent row per position, all of them different colours because nothing ties them together. Right: the relative bias matrix, which is constant along every diagonal because the bias depends only on i − j. Slide the offset to watch a single diagonal light up. That diagonal structure is the seed of everything in Chapter 5.

Hand Calculation: Which Bucket?

Buckets are where relative bias gets its length behaviour, so it is worth computing a few by hand. Take T5's defaults: 32 buckets, split 16 per direction, the first 8 offsets each getting their own bucket, and a maximum distance of 128.

Offsets 0 through 7 are exact: offset 3 is bucket 3, no arithmetic. Past 8, the bucket index is 8 + floor( ln(offset/8) / ln(128/8) × 8 ), clamped at 15.

Offsets 100 and 500 land in the same bucket and therefore get the identical bias. That is not a bug, it is the design: past the maximum distance the scheme stops distinguishing distances at all. It is also exactly why bucketed relative bias shows up as a flat row in the extrapolation arena in Chapter 10, neither collapsing nor improving. It has stopped answering the question rather than answering it badly.

The Cost: A Bias Matrix Per Layer

Relative bias is not free. The score matrix is (L, L) per head, and the bias has to be materialised at the same shape before the softmax. At L = 8192 that is 67 million entries per head, which is exactly the tensor FlashAttention works so hard not to materialise. Shaw-style relative encodings that add vectors to the keys are worse still: they turn one matrix multiply into a gather plus a second multiply.

python
import math, torch, torch.nn as nn

class T5RelativeBias(nn.Module):
    def __init__(self, n_heads, n_buckets=32, max_dist=128):
        super().__init__()
        self.b = nn.Embedding(n_buckets, n_heads)   # (32, H)
        self.n_buckets, self.max_dist = n_buckets, max_dist

    def bucket(self, rel):                       # rel: (L, L) int
        n = self.n_buckets // 2                     # half for each sign
        sign = (rel > 0).long() * n
        r = rel.abs()
        exact = n // 2                              # small gaps get own bucket
        big = exact + (
            torch.log(r.float().clamp(min=1) / exact)
            / math.log(self.max_dist / exact) * (n - exact)
        ).long()
        return sign + torch.where(r < exact, r, big.clamp(max=n - 1))

    def forward(self, L):
        pos = torch.arange(L)
        rel = pos[None, :] - pos[:, None]           # (L, L) = j - i
        return self.b(self.bucket(rel)).permute(2, 0, 1)  # (H, L, L)

Read the shapes. The learned parameter is (32, H), tiny. The thing you add to the logits is (H, L, L), enormous. That gap between parameter cost and runtime cost is the reason the field went looking for something that could be relative without building an L by L tensor. That something is RoPE.

Chapter 3: RoPE, Position as Rotation

Su et al. (2021) asked a sharper question than anyone before them. Instead of adding something to the vectors or to the score, what if we rotate the query and the key by an angle proportional to their positions?

Picture the hands of a clock. Take a 2-D slice of the query vector and turn it by m·θ radians, where m is the query's position. Take the same 2-D slice of the key and turn it by n·θ. Now take their dot product.

Hand Calculation: One Plane, 30 Degrees

Work entirely in 2-D first. Let q = [1, 0] and k = [0, 1], so the angle between them starts at 90 degrees and their dot product is 0. Let θ = 30 degrees = 0.5236 radians.

The 2-D rotation matrix by angle φ is

Case 1: q at position m = 2, k at position n = 1.

Query angle 2·30 = 60 degrees. cos 60 = 0.500, sin 60 = 0.866. R(60)·[1, 0] = [0.500, 0.866].

Key angle 1·30 = 30 degrees. cos 30 = 0.866, sin 30 = 0.500. R(30)·[0, 1] = [−0.500, 0.866].

Dot product: 0.500·(−0.500) + 0.866·0.866 = −0.250 + 0.750 = 0.500.

Case 2: q at position m = 7, k at position n = 6. Same offset, wildly different absolute positions.

Query angle 210 degrees: R(210)·[1, 0] = [−0.866, −0.500]. Key angle 180 degrees: R(180)·[0, 1] = [0.000, −1.000].

Dot product: (−0.866)·0.000 + (−0.500)·(−1.000) = 0.500. Identical.

Hand Calculation: Four Dimensions, Two Planes

Real RoPE splits a d-dimensional head into d/2 independent planes and gives each one its own frequency. Let d = 4, so two planes, with the usual base 10000: θ₀ = 1 and θ₁ = 0.01, the same pair we computed in Chapter 1.

Plane A, frequency 1. Query angle 3·1 = 3 rad. cos 3 = −0.990, sin 3 = 0.141. R(3)·[1, 0] = [−0.990, 0.141]. Key angle 1·1 = 1 rad. cos 1 = 0.540, sin 1 = 0.841. R(1)·[0, 1] = [−0.841, 0.540]. Partial dot: (−0.990)(−0.841) + (0.141)(0.540) = 0.833 + 0.076 = 0.909.

Plane B, frequency 0.01. Query angle 0.03 rad, key angle 0.01 rad. R(0.03)·[1, 0] = [1.000, 0.030]. R(0.01)·[0, 1] = [−0.010, 1.000]. Partial dot: (1.000)(−0.010) + (0.030)(1.000) = −0.010 + 0.030 = 0.020.

Total score = 0.909 + 0.020 = 0.929.

Now check the prediction. The offset is n − m = −2, so plane A should give q^TR(−2)k with the unrotated q and k.

R(−2)·[0,1] = [sin 2, cos 2] = [0.909, −0.416], dotted with [1, 0] gives 0.909. Correct. Plane B: R(−0.02)·[0,1] = [0.020, 1.000], dotted with [1, 0] gives 0.020. Correct.

The four dimensional score decomposed into two independent planar scores, each a function of the offset alone. That is what "d/2 planes" buys you.

See It: Rotate and Watch the Score

Move both position sliders. The left panel shows the rotated query and key in one plane. The right panel plots the score as a function of the offset m − n, with a marker on your current pair. Slide m and n together and the marker does not move, because the score genuinely only sees the difference.

The Same Rotation, Three Ways

Ladder one of three. Identical inputs, identical printed output, from plain Python up to a production PyTorch layer.

python  ·  rung 1: plain Python, no libraries
import math

def rope_plain(v, pos, base=10000.0):
    d = len(v)
    out = [0.0] * d
    for i in range(d // 2):
        theta = 1.0 / (base ** (2.0 * i / d))
        ang = pos * theta
        c, s = math.cos(ang), math.sin(ang)
        x, y = v[2*i], v[2*i + 1]        # the i-th plane
        out[2*i]     = x * c - y * s
        out[2*i + 1] = x * s + y * c
    return out

q = rope_plain([1., 0., 1., 0.], 3)
k = rope_plain([0., 1., 0., 1.], 1)
print(round(sum(a*b for a, b in zip(q, k)), 4))   # 0.9288

python  ·  rung 2: NumPy, whole sequence at once
import numpy as np

def rope_numpy(x, pos, base=10000.0):
    # x: (L, d) with d even.  pos: (L,)
    d = x.shape[-1]
    inv = 1.0 / (base ** (np.arange(0, d, 2) / d))   # (d/2,)
    ang = pos[:, None] * inv[None, :]              # (L, d/2)
    c, s = np.cos(ang), np.sin(ang)
    xe, xo = x[:, 0::2], x[:, 1::2]                # even / odd halves of each plane
    out = np.empty_like(x)
    out[:, 0::2] = xe * c - xo * s
    out[:, 1::2] = xe * s + xo * c
    return out

q = rope_numpy(np.array([[1., 0., 1., 0.]]), np.array([3]))
k = rope_numpy(np.array([[0., 1., 0., 1.]]), np.array([1]))
print(round(float(q @ k.T), 4))                  # 0.9288

python  ·  rung 3: PyTorch, batched and cached
import torch

class RoPE(torch.nn.Module):
    def __init__(self, dim, max_len=8192, base=10000.0):
        super().__init__()
        inv = 1.0 / (base ** (torch.arange(0, dim, 2).float() / dim))
        ang = torch.outer(torch.arange(max_len).float(), inv)  # (L, d/2)
        self.register_buffer("cos", ang.cos(), persistent=False)
        self.register_buffer("sin", ang.sin(), persistent=False)

    def forward(self, x, offset=0):
        # x: (B, H, L, d) -> same shape, rotated in place per plane
        L = x.shape[-2]
        c = self.cos[offset:offset+L][None, None]    # (1,1,L,d/2)
        s = self.sin[offset:offset+L][None, None]
        xe, xo = x[..., 0::2], x[..., 1::2]
        out = torch.empty_like(x)
        out[..., 0::2] = xe * c - xo * s
        out[..., 1::2] = xe * s + xo * c
        return out

rope = RoPE(4)
q = rope(torch.tensor([[[[1., 0., 1., 0.]]]]), offset=3)
k = rope(torch.tensor([[[[0., 1., 0., 1.]]]]), offset=1)
print(round((q * k).sum().item(), 4))               # 0.9288

All three print 0.9288, which is the 0.929 we computed by hand. The offset argument in rung three is not decoration: during generation you rotate one new token at a time, and the offset is its absolute index in the running sequence.

Chapter 4: ALiBi, the Additive Ramp

Press, Smith and Lewis (2021) went the other way. Do not touch the vectors at all. Leave q and k exactly as the projections produced them, and simply subtract a penalty from the attention logit that grows with distance.

That is the entire method. One scalar m_h per head, fixed, not learned. The name is Attention with Linear Biases, and the whole paper is that formula plus the observation that it extrapolates better than anything that came before it.

Why Every Head Gets a Different Slope

If every head had the same slope, every head would have the same field of view, and the model could not simultaneously look at the last three tokens and the whole document. So ALiBi assigns slopes as a geometric sequence. For 8 heads:

Head 1 has slope 0.5, so a token 20 back gets a penalty of 10 logits, which after softmax is a factor of e⁻¹⁰ ≈ 0.000045. That head is effectively blind past a handful of tokens: it is a local syntax head. Head 8 has slope 0.0039, so the same token 20 back is penalised by 0.078 logits, almost nothing. That head sees the whole document.

For n heads the general rule is a geometric ratio of 2^−8/n, chosen so the slope range stays roughly the same whether you have 8 heads or 64.

Hand Calculation: One Bias Row

Take head 2, slope m = 0.25, and a causal query at position i = 4. The bias added to each key position j:

Now suppose the raw scaled dot products for that row are all equal, say 2.00 across the board, so content carries no preference at all. Add the bias: (1.00, 1.25, 1.50, 1.75, 2.00). Exponentiate: (2.718, 3.490, 4.482, 5.755, 7.389), which sum to 23.834. The attention weights are:

Purely from geometry, the nearest token gets 2.7 times the weight of the furthest. Ratios between adjacent weights are all e^0.25 = 1.284. The bias turned a flat content signal into a clean exponential recency profile, and it did so with one number per head and zero parameters trained.

See It: Slopes Across Heads

The heatmap is the causal bias matrix for one head: rows are query positions, columns are key positions, and the upper triangle is masked. Change the head and watch the visible band tighten or widen. The curve underneath is the resulting attention weight as a function of distance, which is where the geometric decay becomes obvious.

ALiBi From Scratch

python
import math, torch

def alibi_slopes(n_heads):
    # geometric sequence with ratio 2 ** (-8 / n_heads)
    start = 2.0 ** (-8.0 / n_heads)
    return torch.tensor([start ** (h + 1) for h in range(n_heads)])

def alibi_bias(n_heads, L):
    pos = torch.arange(L)
    rel = (pos[None, :] - pos[:, None]).float()      # (L, L), j - i
    rel = rel.masked_fill(rel > 0, float("-inf"))       # causal mask
    return alibi_slopes(n_heads)[:, None, None] * rel   # (H, L, L)

b = alibi_bias(8, 5)
print(alibi_slopes(8))          # [0.5, 0.25, 0.125, ..., 0.0039]
print(b[1, 4])                  # [-1.00, -0.75, -0.50, -0.25, 0.00]

Row b[1, 4] is the hand-computed row above, to the digit. In a real kernel you never build b; you recompute slope * (j - i) from the tile indices, which is why ALiBi's memory overhead is exactly zero.

Chapter 5: What Encodings Are Even Possible?

We now have four working methods and no idea whether there are four hundred more waiting to be discovered. In April 2026 Alok Puranik, an ML researcher at Jane Street, spent a Friday afternoon on exactly that question and published the answer: Using group theory to explore the space of positional encodings for attention.

His framing, in his words: "what positional encodings are even possible? As I started investigating, I discovered to my surprise that the space is quite constrained." This chapter and the two after it walk his derivation. It needs nothing you do not already have.

Step 1: Write Down What a PE Even Is

We have a time series of queries q(t) and a time series of keys k(s). Time is written as a continuous variable so the argument covers irregularly sampled series and learned notions of time as in the Mamba family, but you can read t and s as plain integer indices if you prefer.

Without any encoding, attention computes the inner product q(t)^Tk(s). We want to perturb that so it knows about position. To stay cheap, we do not modify each pair independently. Instead we transform the queries and the keys themselves with two explicitly time-dependent functions F and G:

Everything that follows is about that bold middle object. A positional encoding, under this framing, is the square matrix F(t)^TG(s). Change nothing else and you have changed nothing.

Property 1: Linearity

F(t) is linear in q and G(s) is linear in k. Puranik is candid that this one is not rigorously justifiable: "we're working with a vector space, so it's only right that our encoding should be linear. We'll leave the study of nonlinear encodings to others."

Linearity is what lets us write F and G as square matrices at all. Drop it and the object F(t)^TG(s) is not a matrix and the enumeration is hopeless. Note what it costs: ALiBi, which maps the dot product to q^Tk + m(t − s), is not literally of this form. Hold that thought until Chapter 7, where it comes back through a side door.

Property 2: Translation Invariance

For any times t, s and any shift r:

Shift both tokens later by the same amount and the score does not change. Only relative positions are observable. The practical motivation is the one we hit in Chapter 2: if the absolute index is observable, what do you do when you trained on sequences up to length L and now need to handle a longer one?

Here is the picture that makes it click. Lay out F(t)^TG(s) for every pair (t, s) in a table. Translation invariance says every entry on a given diagonal is the same matrix, because moving one step down and one step right leaves the value alone.

Step 2: Normalise, and Meet A(t)

One more harmless assumption: F(t)^TG(t) = I. At equal times the positional modification simply drops out. This costs nothing, because if we had started with F(t)^TG(t) = M for some fixed M, we could fold M into the keys by redefining k, and we already agreed to ignore time-independent transformations.

Two consequences follow immediately. F(t) and G(t) are inverses of each other, and F(t)^TG(s) depends only on t − s. So define

and the whole table becomes one function of one variable. A(0) sits on the main diagonal, A(1) one below, A(−1) one above.

Worked Example: Filling In the Table

Abstractions are easier to trust once you have watched them hold with real numbers. Pick the simplest non-trivial F and G: two dimensional rotations by 30 degrees per step, so F(t) = R(30t) and G(s) = R(30s).

Then F(t)^TG(s) = R(30t)^TR(30s) = R(30(s − t)), because a rotation's transpose is its inverse and composing rotations adds their angles. Tabulate that for the first three times.

Read the diagonals. The main diagonal is R(0), which is the identity, as our normalisation demanded. One step below is R(−30) everywhere. Two steps below is R(−60) everywhere. Nine cells, three distinct values, and each value is determined by t − s alone.

Now check the composition law numerically. R(−30) has entries 0.866, 0.500 on the top row and −0.500, 0.866 on the bottom. Multiply it by itself:

That is R(−60), which is exactly the entry two steps below the diagonal. A(1)·A(1) = A(2), checked by hand. Nothing about rotations was special here; the same collapse happens for any F and G obeying the two properties. The next chapter explains why.

Property 3: Continuity

A(t) is a continuous function of t. Puranik is blunt about why this is safe to assume: "There are discontinuous functions that satisfy our constraints, but they require the axiom of choice to describe and are far too deranged to implement on a physical computer."

Three properties, in order: linearity, translation invariance, continuity. That is the complete set of assumptions. Everything from here is consequence.

Chapter 6: A One-Parameter Group

We ended the last chapter with two facts about the key-translating matrix A(t):

A family of matrices satisfying exactly that is called a one-parameter group. And one-parameter groups are not merely well studied, they are completely classified. Every one of them has the same shape:

for a single fixed matrix L called the generator. That is the whole answer to "what positional encodings are possible." Pick a generator, and you have picked a positional encoding. There is nothing else to pick.

What exp of a Matrix Even Means

The scalar exponential has a power series that converges for every real x: e^x = 1 + x + x²/2! + x³/3! + ... Matrices can be multiplied and added, so plug one in:

That series converges for every square matrix. And the reason it is the right object here is a single identity: if two matrices commute (and tL and sL always do, being multiples of the same L), then exp(sL)·exp(tL) = exp((s+t)L). The group law is the exponential's defining property, transplanted from numbers to matrices.

Hand Calculation: exp of a Skew Matrix Is a Rotation

Take the smallest interesting generator, the 2 by 2 skew matrix

The trick is to compute its powers first. L² = −I. Therefore L³ = L²·L = −L, and L⁴ = I, and the cycle repeats with period 4. Substitute into the series and sort the terms into the ones carrying I and the ones carrying L:

Written out, that is exactly the rotation matrix:

Check it numerically at t = 0.5236 (30 degrees). Term by term, starting from I:

Five terms and we are at cos 30 = 0.8660 and −sin 30 = −0.5000. RoPE is what you get when the generator is skew symmetric. Not by analogy. By computing an exponential.

Hand Calculation: exp of a Nilpotent Matrix Is a Shear

Now the other interesting small generator, the nilpotent matrix

N² = 0 exactly, so every term past the second in the series vanishes. The exponential is a two-term polynomial:

Apply it to the vector [x, v]: you get [x + tv, v]. Puranik points out that this is not a pathology, it is physics. "Consider a frictionless hockey puck launched from position x with velocity v at time 0. At time t, its new position is x + tv, and its velocity remains v. The time evolution of this simple physical system is governed precisely by exp(tN)."

Verify the group law by hand: [1, s; 0, 1]·[1, t; 0, 1] = [1, s + t; 0, 1]. The offsets add. That is A(s)A(t) = A(s+t), and it is why the puck's position after s then t seconds equals its position after s + t seconds. Chapter 7 turns this exact matrix into ALiBi.

The Third Behaviour: Decay

If the generator has a real eigenvalue c, then on that one-dimensional subspace A(t) acts as multiplication by e^ct. Three cases, and Puranik disposes of them quickly.

Two footnotes worth keeping. First, when a model learns a data-dependent decay factor, the right way to read it through this lens is not "the model changes the rate of decay" but "the model learns how far to advance time." Second, the decaying case only makes sense for causal attention. Without a mask, tokens far in the future would have exponentially increased influence on early queries.

From a Real Matrix to Rotation and Decay

One gap worth closing. Real generators do not usually arrive as neat skew or nilpotent blocks. The blog's route is to diagonalise L, which is convenient because it chops the encoding into non-interacting pieces: the d-dimensional space becomes a direct sum of one dimensional spaces, and on each of them the encoding is just multiplication by a scalar exponential.

There is a wrinkle. Diagonalising a real matrix generally needs a complex basis, and the diagonal entries come out complex. But if L was real, the non-real eigenvalues arrive in conjugate pairs, so we take a slightly coarser decomposition: real eigenvalues get their own one dimensional subspace, and each conjugate pair gets a two dimensional one.

On a conjugate pair, write the eigenvalue in polar form as c + iω. Then on that two dimensional block,

where R is the ordinary rotation matrix. Converting to that form is a real change of basis, which the blog calls the real canonical form, and we already agreed to ignore time-independent basis changes. So a real generator always decomposes into rotations at fixed frequencies, each optionally damped, plus pure decay on the real axes.

The Payoff: One Slider, Every Encoding

This is the showcase. Pick a generator L, drag t, and watch A(t) = exp(tL) do its thing to a vector. The orbit is the path the vector traces as t sweeps. The matrix on the right is A(t) computed from the closed forms above, and the check line multiplies A(t/2) by itself to confirm the group law numerically.

The Awkward Case: Defective Generators

One loose end. The clean classification above assumed L is diagonalisable. Some matrices are not. A defective matrix is one whose eigenvectors do not span the space, and the nilpotent N above is the smallest example: its only eigenvalue is 0, twice, but it has a single eigenvector.

Defective generators are handled by putting L in Jordan normal form, and they behave differently from everything else. Puranik: "unlike diagonalizable matrices which produce only exponential and trigonometric factors, defective matrices give rise to positional encodings involving polynomial terms." We just saw the simplest polynomial, the linear term t in the shear. Bigger Jordan blocks give t², t³ and up.

Keep that in your pocket. It is the seed of both ALiBi and the one genuinely unexplored family, and Chapter 7 opens both.

Chapter 7: The Families

Every positional encoding is exp(tL) for some generator L. Now we sort the generators. A real matrix decomposes into blocks, and there are only three kinds of block, so there are only three families.

Family 1: Rotations

Complex conjugate eigenvalue pairs give 2-D blocks that act as rotation. In real canonical form the block is the skew matrix from Chapter 6, and the exponential is a rotation by a constant angular frequency ω:

With c = 0 that is exactly RoPE, one rotation per plane, one frequency per plane. With c < 0 it is exponentially damped RoPE, the encoding used by RetNet and Mamba-3. With c > 0 it blows up and gets thrown out. Everything in this family preserves or shrinks norms and gives a score that oscillates with offset.

Family 2: Unipotent Ramps

Defective blocks give polynomials. The smallest defective block is the nilpotent N, whose exponential is the shear I + tN. This is where ALiBi lives, and getting it there is a genuinely cute argument.

Puranik notes the objection first. Stefan Gliga pointed out on X that restricting to linear functions of q and k excludes ALiBi, "which applies a penalty to the attention dot product." True, and it looks fatal: ALiBi maps q^Tk to q^Tk + m(t − s), and adding a constant is not a linear map of q. But ALiBi is reachable if we allow ourselves to augment the queries and keys with two extra constant coordinates.

Multiply it out by hand. With u = t − s:

Exactly the ALiBi bias, produced by a legal linear positional encoding generated by a defective matrix. Puranik's dry summary: "So, intentionally or otherwise, Stefan found a useful example of a positional encoding generated by a defective matrix."

Family 3: The Strange Legal Class

The shear was the simplest defective block. Nothing stops L from having a 3 by 3 Jordan block, or a 4 by 4 block built from a doubled conjugate pair of complex eigenvalues. Those produce higher polynomials in t: a score term proportional to t², or to t³, or to t²cos(ωt).

This family is legal under all three properties. It is also, as far as anyone can tell, entirely unused. Puranik: "it's entirely unclear how to interpret these polynomials in the context of attention. I failed to find any existing literature which directly addresses the possibility of defective generators for positional encodings in deep learning, and most likely they have no practical application. Nevertheless, the curious possibility remains."

Notice why it is unattractive rather than illegal. A t² term means the influence of a key changes quadratically with the gap, so either it explodes faster than any softmax can absorb, or, if the sign points the other way, it suppresses everything past a short horizon far harder than a linear ramp does. Neither is obviously a good prior for language. But nobody has actually tried it.

See It: Three Families, One Axis

Each curve is the perturbation a family adds to the attention score as the offset grows. Rotation oscillates and never dies. Damped rotation oscillates inside a shrinking envelope. The unipotent ramp is a straight line, which after softmax is geometric decay. The polynomial curls away from all of them.

The Whole Space, On One Page

Chapter 8: GRAPE, Rotations You Can Learn

The blog proves the space of positional encodings is small. A paper published at ICLR 2026 builds the same object on purpose and puts learnable parameters in it: Group Representational Position Encoding by Yifan Zhang, Zixiang Chen, Yifeng Liu, Zhen Qin, Huizhuo Yuan, Kangping Xu, Yang Yuan, Quanquan Gu and Andrew Chi-Chih Yao. GRAPE for short.

GRAPE's central object is exactly the one we derived:

with n the position, ω a frequency, and L the generator. Two choices of L give the two halves of the framework. A skew-symmetric L in so(d) gives Multiplicative GRAPE, rotations in the special orthogonal group SO(d), which is this chapter. A nilpotent L gives Additive GRAPE, unipotent actions in the general linear group GL, which is the next.

The Rank-2 Generator

The smallest non-trivial skew generator rotates inside a single plane, and you build it from the two vectors that span that plane. For a, b in d dimensions:

Check it is skew: transpose the whole thing and you get b a^T − a b^T = −L. So L lies in so(d), the Lie algebra of skew-symmetric matrices, which is exactly the set of generators whose exponentials are rotations. Define the plane scalars

Δ is the squared area of the parallelogram a and b span, so s is that area. It will turn out to be the rotation angle per unit of nω, which is a satisfying thing for an area to be.

The One Fact That Makes It Cheap

Apply L twice and something clean happens. GRAPE's key algebraic observation:

where P_U is the orthogonal projector onto the plane U. In words: squaring L flattens everything onto the action plane and scales by −s². Its eigenvalues are ±is and then zero, d − 2 times, and its minimal polynomial is λ(λ² + s²).

Because the minimal polynomial has degree three, every power of L past the second reduces to a combination of I, L and L². So the infinite exponential series collapses to three terms. This is the Rodrigues formula, the same one used for 3-D rotations in robotics since 1840:

Substituting L² = −s²P_U makes the geometry explicit:

Hand Calculation: A Plane RoPE Cannot Reach

Take d = 4 and pick the plane spanned by coordinates 1 and 3, which is exactly what canonical RoPE cannot do, since RoPE always pairs (1,2) and (3,4).

Rotate the vector x = (1, 0, 0, 0), which is a itself. Two inner products first: u = a·x = 1 and v = b·x = 0.

Now L·x = a(b·x) − b(a·x) = a·0 − b·1 = −b = (0, 0, −1, 0). And P_Ux = x, because x lies in the plane.

Plug into the geometric form with angle nωs = 0.5236:

Norm check: 0.866² + 0.500² = 0.750 + 0.250 = 1.000. Preserved exactly, as an SO(d) element must. Coordinates 2 and 4 never moved. A learned generator just coupled dimension 1 with dimension 3 by rotation, and canonical RoPE has no way to express that.

Cost: O(d), Never Materialised

GRAPE never builds the d by d matrix G(n). Applying it to a vector x is:

with f₁ = sin(nωs)/s and f₂ = (1 − cos(nωs))/s², guarded by their Taylor series as s approaches 0. Two inner products and a handful of scaled vector adds: O(d) flops per head, with a small constant.

Run our hand example through it as a check. u = 1, v = 0, γ = 0, α = β = 1, f₁ = 0.5000, f₂ = 0.1340. y = x + 0.5(a·0 − b·1) + 0.134[0 − a·1 − 0] = x − 0.5b − 0.134a = (0.866, 0, −0.5, 0). The same answer.

The b = J a Shortcut

Let J be the canonical 90-degree rotation, block diagonal with J^T = −J and J² = −I. Setting b = J a means "b is a rotated a quarter turn." Then γ = a^TJ a = 0 and β = α, so s = α and the whole thing simplifies to a pure planar rotation by angle nωα. If you normalise ‖a‖ = 1, the angle is just nω. The paper calls this frequency-norm coupling: without normalisation the effective frequency is ω‖a‖², so the scale of a can always be absorbed into ω.

Multi-Subspace: RoPE Falls Out

One rank-2 generator touches one plane and leaves d − 2 dimensions alone. To cover the whole head, sum d/2 of them on mutually orthogonal planes. Orthogonal planes commute, so their exponentials multiply cleanly:

Stare at the right-hand side. A block diagonal product of planar rotations with per-plane angles nθ_i. That is the definition of RoPE. The paper states it as Proposition 3.1: choose d/2 mutually orthogonal vectors a_i, set b_i = J a_i, and the commuting product equals the standard RoPE map in a possibly learned orthogonal basis. Make the planes the canonical coordinate pairs and the frequencies log-uniform, and you recover canonical RoPE exactly.

Letting the Planes Interact

Commuting planes never talk to each other. To let them, GRAPE uses a thin compression: write L = E L_r E^T with E an orthonormal d by r frame and L_r a small r by r skew matrix. The spectrum of L is the spectrum of L_r padded with zeros, so the expressive power of a non-commuting mixture is fully captured by r/2 mode angles. Cost rises to O(rd) per head, still linear in d.

The reason to bother: cross-subspace feature coupling. In canonical RoPE, dimension 5 can never be rotated into dimension 40 no matter what the data says. With a learned or non-commuting generator it can.

Inside Multi-Head Attention

Concretely, with H heads and per-head width d, the operator G_h(t) in SO(d) is applied to the query and the key of head h at position t, and the value is left alone:

The logit is the usual scaled dot product of the transformed pair. Because G_h comes from a one-parameter subgroup, G_h(t)^TG_h(j) equals G_h(j − t) and the logit depends only on the offset. Every head can carry its own generator, so one head can rotate slowly in a learned plane while another rotates fast in a different one.

Two shape facts worth holding onto. The learned parameters are two d-vectors per head, so 2·H·d numbers, which for a 32-head model with d = 128 is 8,192 numbers per layer. Nothing about the attention output projection, the value path, or the residual stream changes. A GRAPE-M layer is a drop-in replacement for a RoPE layer.

See It: Tilt the Plane

The grid shows a 4-D head as its two coordinate pairs. RoPE mode rotates inside the fixed pairs and the cross terms stay dark. Learned mode tilts the action plane so a rotation in one plane leaks into another dimension, which is the cross-subspace coupling the paper is after. The bars show the output vector.

The Same Group Action, Three Ways

Ladder two of three: the general exp(nωL) action for a rank-2 generator, closed form, no scipy.linalg.expm anywhere. All three rungs print the hand-computed vector from earlier in the chapter.

python  ·  rung 1: plain Python, no libraries
import math

def grape_m_plain(x, a, b, n, omega):
    dot = lambda p, q: sum(pi*qi for pi, qi in zip(p, q))
    al, be, ga = dot(a, a), dot(b, b), dot(a, b)
    s = math.sqrt(max(al*be - ga*ga, 0.0))
    th = n * omega * s
    if s < 1e-8:                          # series guard as s -> 0
        f1, f2 = n * omega, 0.5 * (n*omega)**2
    else:
        f1, f2 = math.sin(th) / s, (1 - math.cos(th)) / (s*s)
    u, v = dot(a, x), dot(b, x)
    return [
        x[i]
        + f1 * (a[i]*v - b[i]*u)
        + f2 * (ga*(a[i]*v + b[i]*u) - be*a[i]*u - al*b[i]*v)
        for i in range(len(x))
    ]

a = [1., 0., 0., 0.]; b = [0., 0., 1., 0.]
print([round(z, 3) for z in grape_m_plain([1.,0.,0.,0.], a, b, 1, 0.5236)])
# [0.866, 0.0, -0.5, 0.0]   the hand calculation, exactly

python  ·  rung 2: NumPy, a whole sequence of positions
import numpy as np

def grape_m_numpy(x, a, b, n, omega):
    # x: (L, d)   n: (L,)   a, b: (d,)
    al, be, ga = a @ a, b @ b, a @ b
    s = np.sqrt(max(al*be - ga*ga, 0.0))
    th = n * omega * s                          # (L,)
    f1 = np.where(s < 1e-8, n*omega, np.sin(th) / max(s, 1e-8))
    f2 = np.where(s < 1e-8, 0.5*(n*omega)**2, (1 - np.cos(th)) / max(s*s, 1e-16))
    u = x @ a; v = x @ b                        # (L,) each
    return (x
        + f1[:, None] * (np.outer(v, a) - np.outer(u, b))
        + f2[:, None] * (ga*(np.outer(v, a) + np.outer(u, b))
                        - be*np.outer(u, a) - al*np.outer(v, b)))

x = np.array([[1., 0., 0., 0.]])
print(grape_m_numpy(x, np.array([1.,0.,0.,0.]),
                   np.array([0.,0.,1.,0.]), np.array([1.]), 0.5236).round(3))
# [[ 0.866  0.    -0.5    0.   ]]

python  ·  rung 3: PyTorch, learnable a and b, batched heads
import torch, torch.nn as nn

class GrapeM(nn.Module):
    """One rank-2 learned plane per head. No matrix is ever built."""
    def __init__(self, n_heads, dim, omega=1.0):
        super().__init__()
        self.a = nn.Parameter(torch.randn(n_heads, dim) / dim**0.5)
        self.b = nn.Parameter(torch.randn(n_heads, dim) / dim**0.5)
        self.omega = omega

    def forward(self, x, pos):
        # x: (B, H, L, d)   pos: (L,)
        a, b = self.a[None, :, None, :], self.b[None, :, None, :]
        al = (a*a).sum(-1, keepdim=True)
        be = (b*b).sum(-1, keepdim=True)
        ga = (a*b).sum(-1, keepdim=True)
        s  = (al*be - ga*ga).clamp(min=1e-12).sqrt()      # (1,H,1,1)
        th = pos[None, None, :, None] * self.omega * s     # (1,H,L,1)
        f1, f2 = th.sin() / s, (1 - th.cos()) / (s*s)
        u = (x*a).sum(-1, keepdim=True)                  # (B,H,L,1)
        v = (x*b).sum(-1, keepdim=True)
        return (x + f1*(a*v - b*u)
                  + f2*(ga*(a*v + b*u) - be*a*u - al*b*v))

Shapes to notice in rung three. The learned parameters are two vectors per head, 2·H·d numbers total, against LieRE's H·d². The forward pass allocates nothing bigger than x itself. And because G(n)^TG(n) = I by construction, the rotated keys can be written straight into a KV cache and never touched again, which is the streaming property Chapter 9 makes explicit.

Chapter 9: GRAPE, Ramps Are Group Actions Too

ALiBi adds a number to the logit. That does not look like a group action at all: group actions multiply, biases add. GRAPE's second half shows the appearance is wrong, using the same trick computer graphics has used since the 1970s.

The Homogeneous Lift

A translation is not a linear map. Shifting every point by a fixed vector moves the origin, and linear maps must fix the origin. The standard fix is to add one coordinate, pin it to 1, and do the translation as a shear in the larger space.

Read the middle line. A is nilpotent, so its exponential truncates after two terms exactly as in Chapter 6. The result is unipotent, meaning all of its eigenvalues equal 1. And it is still a one-parameter group: G_add(n + m) = G_add(n)G_add(m), because the nωu terms simply add.

Applying It, and Why the Inverse Transpose

GRAPE applies the group element to the query and the inverse transpose to the key:

Then q̃_i^Tk̃_j = q̂_i^T G_add(j − i)^−T k̂_j, which depends only on j − i. In SO(d) the transpose is the inverse, so this pairing is invisible in Chapter 8; in GL it is not, and you have to write the inverse transpose explicitly to keep the relative law exact.

Exact ALiBi as a Rank-1 Unipotent in GL(d+2)

Now recover ALiBi on the nose. Augment with two constant coordinates, one carried by the query and one by the key:

The lift is asymmetric on purpose: the extra coordinates are 1 and 0 for the query, 0 and 1 for the key, so the base inner product is unchanged, q̂^Tk̂ = q^Tk with no extra constant. Then G_add,h(m)^−T = I − m A_h^T and the score is

Hand Calculation: The Chapter 4 Row, Rebuilt from the Lift

Take head 2 again, β = 0.25, query at i = 4, and suppose every raw score q^Tk equals 2.00 as before. Only one entry of the rank-1 update ever fires: the query supplies a 1 in coordinate d+1, the key supplies a 1 in coordinate d+2, and every other product is zero. So the extra term is exactly (j − i)β.

Identical to the ALiBi row we computed in Chapter 4, digit for digit. In causal attention j is never greater than i, so m is never positive and the term is always a penalty, never a reward.

Making the Slope Content-Dependent

ALiBi's slope is a constant per head. GRAPE lets the tokens choose it, without breaking anything, by scaling the same nilpotent basis with non-negative gates:

The paper calls this GRAPE-A-QK. Because both gates scale the same nilpotent basis, the generators still commute, so the group structure and the exact relative law survive. The softplus guarantees the rate is non-negative, so with j at most i the term is always a penalty. Two learned vectors u and v per head buy a slope the model can adapt per token pair.

FoX Falls Out Too

The Forgetting Transformer learns a per-head, per-token forget gate f_t in (0, 1] and adds the cumulative log-gate to the logits: D_ij = ∑_ℓ=j+1ⁱ log f_ℓ. Set ω_t = log f_t and the additive bias b(t, j) = ∑_ℓ=j+1^t ω_ℓ coincides with FoX's bias exactly. GRAPE proves FoX is an exact instance of the path-integral additive case with endpoint-independent edges, and when the gates are constant per head, FoX reduces to exact ALiBi.

Streaming: What the KV Cache Sees

The reason all of this is practical is a cache policy that never rewrites. Because the group element attached to a token depends only on that token's own position, you transform the key once, when the token arrives, and store the transformed vector:

When t increments, nothing in the cache needs touching. The group law does the bookkeeping: G(t)^TG(j) is automatically G(j − t). The complexity matches RoPE, and the same policy holds for the additive family.

See It: Write Once, Read Many

Press play. Tokens arrive on the left and each key is transformed exactly once at write time, then frozen (the lock appears). The query at the current step is transformed and dotted against the frozen cache. Switch to the "needs rewrite" mode to see what a non-group scheme would cost: every cached key lights up again on every step.

Composing the Two Halves

Nothing forces you to choose between rotating and biasing. The two families compose, and GRAPE gives two equivalent ways to say so.

The easy way is at the logit level. Rotate the query and key with a multiplicative generator, then add the additive bias to the resulting score:

The rigorous way is a single group action in GL(d+2): take the direct sum of the skew generator acting on the original d dimensions and the gated nilpotent generator acting on the two lifted coordinates. Because they live on disjoint coordinate blocks they commute, so the exponential of the sum is the product of the exponentials, and the composed map is still a one-parameter group with the same exact relative law and the same write-once cache policy.

The Same Bias, Three Ways

Ladder three of three: the additive family, from the definition to a production layer. All three print the hand table above.

python  ·  rung 1: plain Python, straight from the lift
def additive_score_plain(q, k, i, j, beta):
    # homogeneous lift: q_hat = [q, 1, 0], k_hat = [k, 0, 1]
    q_hat = list(q) + [1.0, 0.0]
    k_hat = list(k) + [0.0, 1.0]
    m = j - i
    # G(m)^-T = I - m * A^T, with A^T = -beta * e_{d+1} e_{d+2}^T
    # so the ONLY extra term is q_hat[d] * (m * beta) * k_hat[d+1]
    base  = sum(x*y for x, y in zip(q_hat, k_hat))
    extra = q_hat[-2] * (m * beta) * k_hat[-1]
    return base + extra

q = [1.0, 1.0]; k = [1.0, 1.0]        # raw dot product = 2.00
print([round(additive_score_plain(q, k, 4, j, 0.25), 2) for j in range(5)])
# [1.0, 1.25, 1.5, 1.75, 2.0]

python  ·  rung 2: NumPy, the whole causal matrix
import numpy as np

def additive_bias_numpy(L, slopes):
    pos = np.arange(L)
    m = pos[None, :] - pos[:, None]              # (L, L) = j - i
    bias = slopes[:, None, None] * m[None]         # (H, L, L)
    return np.where(m[None] > 0, -np.inf, bias)   # causal

scores = np.full((1, 5, 5), 2.0)
out = scores + additive_bias_numpy(5, np.array([0.25]))
print(out[0, 4])                                # [1.   1.25 1.5  1.75 2.  ]

python  ·  rung 3: PyTorch, content-gated slope (GRAPE-A-QK)
import torch, torch.nn as nn, torch.nn.functional as F

class GrapeA(nn.Module):
    def __init__(self, n_heads, dim, omega=1.0):
        super().__init__()
        self.v = nn.Parameter(torch.zeros(n_heads, dim))   # query gate
        self.u = nn.Parameter(torch.zeros(n_heads, dim))   # key gate
        self.omega = omega

    def forward(self, q, k, logits):
        # q, k: (B, H, L, d)   logits: (B, H, L, L)
        d = q.shape[-1]
        lam_q = F.softplus((q * self.v[None, :, None, :]).sum(-1) / d**0.5)
        lam_k = F.softplus((k * self.u[None, :, None, :]).sum(-1) / d**0.5)
        L = q.shape[-2]
        pos = torch.arange(L, device=q.device)
        m = (pos[None, :] - pos[:, None]).float()          # (L, L)
        rate = lam_q[..., :, None] + lam_k[..., None, :]      # (B, H, L, L)
        return logits + self.omega * m * rate               # penalty, m <= 0

Set v = u = 0 in rung three and softplus(0) = 0.693 for both gates, giving a constant slope: you are back at ALiBi with β = 1.386ω. The learned version is a strict superset, which is the entire pitch of the additive half of GRAPE.

Chapter 10: Length Extrapolation

Everything so far has been about correctness at training length. This chapter is about the thing that actually breaks in production: you trained at 4,096 tokens and a user pastes 60,000.

Why RoPE Breaks, Precisely

RoPE is relative by construction, so it should extrapolate. It does not, and the reason is not the mechanism, it is the training distribution.

Take a Llama-shaped head: d = 128, so 64 planes, base 10,000. The slowest plane is i = 63, with θ = 1 / 10000^126/128 = 1 / 8660 = 0.0001155 radians per token.

That is the failure. Not that the formula stops working, but that the low-frequency planes present rotation angles that are out of distribution. The dot product formula is still exact; the weights that consume it were never fit on those inputs.

Compare the fastest plane, i = 0, with θ = 1. It completes a full turn every 6.28 tokens, so across 4,096 tokens it wraps 652 times. Every possible angle appeared thousands of times in training. High frequencies are saturated with data; low frequencies are starved. Any fix has to treat them differently, and every good fix does.

See It: The Arena

Each row is a method, each column an evaluation length. Colour is modelled perplexity quality relative to the training length: teal is holding up, warm is degrading, red is broken. The vertical line marks where training stopped. Drag it and watch the picture reorganise.

What the Arena Shows

Two lessons sit in that table. First, being relative is necessary but not sufficient: sinusoidal and RoPE both encode relativity and both still degrade, because the values they present go out of range. Second, methods that clamp or saturate (ALiBi, bucketed bias) extrapolate gracefully precisely because they throw information away past a horizon.

Fix 1: Position Interpolation

Chen et al. (2023) had the blunt idea. If position 32,768 is out of range, lie about it. Divide every position by the extension factor s so that a 32,768-token sequence presents positions in the range the model already knows.

It works, with a cost. Every plane gets squeezed, including the fast ones that were perfectly happy. Adjacent tokens that used to be 1 radian apart in the fast plane are now 1/8 of a radian apart, so the model's ability to tell neighbours apart degrades. Position interpolation needs fine-tuning to recover, typically a thousand steps or so.

Fix 2: NTK-Aware Base Scaling

The insight that makes interpolation nearly free: do not scale positions, scale the base. Because the base sets a geometric ladder of frequencies, raising it stretches the slow planes hard and the fast planes almost not at all, which is exactly the treatment the analysis above asked for.

Check both ends of the ladder.

That last row is the whole trick. The slowest plane got divided by almost exactly the extension factor, so over 32,768 tokens it now sweeps 32768 × 0.0000145 = 0.475 radians, the same 27 degrees it saw during training. The fastest plane kept its full resolution. Nothing is out of distribution and nothing lost precision.

Fix 3: YaRN, Per Frequency

YaRN (Peng et al., 2023) makes the treatment explicit instead of implicit. Sort the planes by how many full cycles they complete inside the training context, then apply three different policies:

YaRN adds one more piece: a temperature on the attention logits, because spreading positions changes the expected magnitude of the scores and the softmax needs rescaling to keep its entropy where training left it.

python  ·  the YaRN per-frequency ramp
import math, torch

def yarn_inv_freq(dim, base, scale, L_train,
                   alpha=1.0, beta=32.0):
    """alpha, beta: how many full turns a plane must complete inside
    L_train before we stop interpolating it. Below alpha turns the plane
    is starved and gets the full 1/scale; above beta it is saturated and
    is left alone; in between it ramps."""
    i = torch.arange(0, dim, 2).float()
    inv = 1.0 / (base ** (i / dim))            # (d/2,) untouched freqs
    turns = L_train * inv / (2 * math.pi)         # cycles inside training ctx

    # ramp: 0 where starved (interpolate fully), 1 where saturated
    ramp = ((turns - alpha) / (beta - alpha)).clamp(0.0, 1.0)

    interp = inv / scale                            # squeeze by the factor
    return interp * (1 - ramp) + inv * ramp        # blend, per plane

f = yarn_inv_freq(128, 10000.0, scale=8.0, L_train=4096)
print(float(f[0]), float(f[-1]))
# 1.0                    fastest plane: untouched, 652 turns in training
# 1.4435e-05             slowest plane: divided by 8, it never finished a turn

Read the two printed numbers against the hand calculation above. The fastest plane keeps its frequency of 1.0 exactly, because it completed 652 turns during training and needs no help. The slowest plane drops from 0.0001155 to 0.00001444, a factor of 8, because it swept only 27 degrees and every angle past that is unseen. The ramp decides where the boundary sits, and the two constants alpha and beta are the only tuning YaRN really asks for.

python  ·  the entire NTK fix
def ntk_base(base, dim, scale):
    # scale = new_context / train_context, e.g. 32768 / 4096 = 8
    return base * scale ** (dim / (dim - 2))

print(round(ntk_base(10000, 128, 8)))       # 82685

# In practice this is one line in the model config:
#   "rope_theta": 82685
# followed by a few hundred fine-tuning steps on long documents.

Chapter 11: Choosing and Shipping a PE

Theory says the space is small. Engineering says pick a point in it and live with the consequences. This chapter is the decision, the checklist, and the list of things that actually go wrong.

The Picker

Answer three questions and read the recommendation. This is not a flowchart of boxes; it is the actual decision, with the reason and the runner-up, because the runner-up is what you switch to when the first choice breaks.

The Checklist

One row per method. Read the second column first: it is the discriminating property that decides everything else.

What GRAPE Actually Measured

Numbers matter more than frameworks, so here is exactly what the paper reports and nothing else. All of it is from Section 6.

Setup (Section 6.1). The nanoGPT codebase with a Llama model, changing only the positional encoding. FineWeb-Edu 100B, of which 50B tokens were used for training. Context length 4,096, batch size 480, QK RMSNorm for stability, AdamW with a cosine schedule and 2,000 warm-up steps, and RoPE base 10,000.

Medium models are 353M parameters with 24 layers, 8 heads and hidden size 1024. Large models are 770M with 36 layers, 10 heads, hidden size 1280 and head dimension 128. Evaluation is 0-shot with lm-evaluation-harness, averaged over seven tasks.

One more observation the paper makes in Section 6.2, visible in its Figure 3: at 770M the RoPE run suffers training instability, while the GRAPE run improves steadily throughout. Stability is a real reason to care about a positional encoding, separate from any benchmark average.

The Gotchas That Actually Bite

A Five-Minute Audit

When you inherit an attention implementation and want to know whether its positional encoding is sound, these five checks find almost every real bug.

Chapter 12: Cheat Sheet and Connections

Everything on one page. Nothing new here, so read it as a reference rather than a lesson.

Symbol Glossary

Every Formula in Plain English

The Full Comparison

The Timeline

Six Things People Get Wrong

Where to Go Next

Transformers for the attention mechanism these encodings modify. Attention Variants for MHA, GQA and FlashAttention, which constrain what a positional encoding is allowed to cost. Vision Transformers for the 2-D case where the coordinate pairing question is unavoidable. State Space Models for the recurrent view, where the damped rotation generator is the state transition itself. Embedding Layers for how token and position embeddings share a vector space in the first place.

Key Sources

• Vaswani et al. "Attention Is All You Need." NeurIPS 2017. arXiv:1706.03762. Sinusoidal encoding.
• Shaw, Uszkoreit, Vaswani. "Self-Attention with Relative Position Representations." NAACL 2018. arXiv:1803.02155.
• Su et al. "RoFormer: Enhanced Transformer with Rotary Position Embedding." 2021. arXiv:2104.09864. RoPE.
• Press, Smith, Lewis. "Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation." ICLR 2022. arXiv:2108.12409. ALiBi.
• Chen et al. "Extending Context Window of Large Language Models via Positional Interpolation." 2023. arXiv:2306.15595.
• Peng et al. "YaRN: Efficient Context Window Extension of Large Language Models." 2023. arXiv:2309.00071.
• Lin et al. "Forgetting Transformer: Softmax Attention with a Forget Gate." 2025. FoX.
• Alok Puranik. "Using group theory to explore the space of positional encodings for attention." Jane Street Tech Blog, 22 April 2026. blog.janestreet.com.
• Zhang, Chen, Liu, Qin, Yuan, Xu, Yuan, Gu, Yao. "Group Representational Position Encoding." ICLR 2026. arXiv:2512.07805. GRAPE.