Music Transformer

Chapter 1: MIDI Event Representation

Before we can feed music to a Transformer, we need to represent it as a sequence of discrete tokens. The Music Transformer uses an event-based MIDI representation developed by Oore et al. (2018). Instead of a piano roll (a 2D grid of time vs pitch), the music is encoded as a 1D sequence of events, like text.

The vocabulary consists of 388 event types across four categories:

A simple example: playing middle C for half a second at medium volume:

A polyphonic texture (multiple notes at once) is a key advantage of this event representation. A chord is represented by consecutive NOTE_ON events with no TIME_SHIFT between them — the model learns that consecutive NOTE_ON events without a TIME_SHIFT form a chord:

This encodes a C major chord (C-E-G, MIDI 60-64-67) held for 1 second at forte volume.

Why this representation works for the Transformer

The event representation has several properties that make it ideal for Transformer modeling:

Handling expressive performance

The VELOCITY events capture dynamics — how loudly or softly notes are played. A skilled pianist varies velocity constantly: a melody note might be played forte (loud, VELOCITY_28) while the accompanying chord is piano (soft, VELOCITY_8). The 32 velocity bins are coarse but sufficient to capture the basic shape of a performance's dynamics. The ordering convention is that a VELOCITY event sets the velocity for all subsequent NOTE_ON events until the next VELOCITY event appears.

TIME_SHIFT events with 10ms granularity capture timing nuance. A note slightly before the beat (anticipation) vs slightly after (laid-back feel) is the difference of 20-30ms — just 2-3 TIME_SHIFT events. This granularity is enough for the model to learn micro-timing patterns that distinguish a mechanical MIDI playback from an expressive human performance.

python
# MIDI event vocabulary
VOCAB_SIZE = 388
# NOTE_ON:     0-127  (128 events)
# NOTE_OFF:  128-255  (128 events)
# TIME_SHIFT: 256-355 (100 events, 10ms to 1s)
# VELOCITY:  356-387  (32 events)

def encode_note_on(pitch):
    return pitch  # 0-127

def encode_note_off(pitch):
    return 128 + pitch  # 128-255

def encode_time_shift(ms):
    # Quantize to 10ms bins, max 1000ms
    bins = min(100, max(1, round(ms / 10)))
    return 256 + bins - 1  # 256-355

def encode_velocity(vel):
    # Quantize 0-127 to 32 bins
    bin_idx = min(31, vel // 4)
    return 356 + bin_idx  # 356-387

# C major chord for 1s at medium velocity:
sequence = [
    encode_velocity(80),      # VELOCITY_20
    encode_note_on(60),       # NOTE_ON_60 (C4)
    encode_note_on(64),       # NOTE_ON_64 (E4)
    encode_note_on(67),       # NOTE_ON_67 (G4)
    encode_time_shift(1000),  # TIME_SHIFT_100
    encode_note_off(60),      # NOTE_OFF_60
    encode_note_off(64),      # NOTE_OFF_64
    encode_note_off(67),      # NOTE_OFF_67
]

Chapter 2: Absolute vs Relative Position

The original Transformer (Vaswani et al., 2017) uses absolute positional encodings: a fixed vector is added to each token embedding that encodes "I am at position t." Sinusoidal encodings use:

This tells the model "this token is at absolute position 42." But consider a musical sequence: a C-E-G chord at positions 10-12, and the same C-E-G chord at positions 200-202. With absolute encodings:

The C at position 10 and the C at position 200 have the same content embedding but completely different positional vectors. The model must learn — separately for every possible offset — that "a C followed by an E" is a third interval. With 2000+ possible positions, this is an enormous learning burden.

Relative position representations (Shaw et al., 2018) replace "I am at position 42" with "the key is 5 positions before me." The attention computation now depends on the distance between query and key, not their absolute locations:

The benefit is especially powerful for transposition. A melody in C major and the same melody in G major are exactly the same pattern, shifted by 7 semitones. With relative pitch encoding, the model doesn't need to relearn the pattern for every possible key — it learns the interval structure once.

Mathematical formalization

Let's be precise. In the standard Transformer, the attention logit between query at position i and key at position j is:

Expanding this product gives four terms:

The last three terms all depend on absolute positions. In the relative formulation, these are replaced by terms that depend only on the offset (i-j). The substitution PE_j → E_r[i-j] converts absolute position dependence to relative.

python
# Absolute vs relative attention

# Absolute: position info added to embeddings
def absolute_attention(X, W_q, W_k, W_v, PE):
    # Add absolute position to input
    X = X + PE  # PE[t] encodes "I am at position t"
    Q, K, V = X @ W_q, X @ W_k, X @ W_v
    scores = Q @ K.T / (d ** 0.5)
    return F.softmax(scores, dim=-1) @ V

# Relative: position info injected into attention scores
def relative_attention(X, W_q, W_k, W_v, E_r):
    # E_r[i-j] encodes "key is (i-j) steps away from query"
    Q, K, V = X @ W_q, X @ W_k, X @ W_v
    # Content-based scores (same as standard)
    content_scores = Q @ K.T
    # Position-based scores (query vs relative position)
    position_scores = Q @ E_r.T  # NEW: relative offset
    scores = (content_scores + position_scores) / (d ** 0.5)
    return F.softmax(scores, dim=-1) @ V

Chapter 3: The Relative Attention Mechanism

Let's formalize relative attention. In standard attention, the score between query at position i and key at position j is:

Shaw et al. (2018) extended this by adding a learned relative position embedding a_ij that depends on the distance (i - j):

Where a_ij = E_r[clip(i-j, -K, K)] is a learned d-dimensional embedding for the relative offset (i-j), clipped to a maximum distance K. The full decomposition (from Dai et al., 2019, which the Music Transformer builds on) expands the attention score into four terms:

Let's understand each term:

The vectors u and v are learned global bias vectors (shared across all queries), replacing the absolute position encoding of the query. E_r is a lookup table of learned embeddings indexed by the relative distance (i-j).

Why four terms, not two?

You might wonder: why not just replace Q·K^T with (Q + position)·(K + position)^T? The decomposition into four terms happens naturally from this expansion, but the key insight is that the position-dependent terms should use different representations than the content-dependent terms. In the full formulation:

These serve different roles: content keys encode what is at a position, while position embeddings encode where it is relative to the query. Separating them lets the model independently learn content-based and position-based attention patterns. A head might learn to attend to "the closest NOTE_ON event" (position-based) regardless of which note it is, or to "any occurrence of middle C" (content-based) regardless of where it appears.

The memory cost problem

Here's the catch: a naive implementation of relative attention requires storing the full L × L × d tensor of relative position embeddings, where L is the sequence length and d is the head dimension. For a 2048-token music sequence with d = 64:

This is O(L²D) memory — much worse than the O(L²) of standard attention. With 8 heads, that's over 2 billion entries just for the relative position embeddings. This made the approach impractical for long sequences... until the skewing trick.

Chapter 4: The Skewing Trick

This is the key technical contribution of the Music Transformer paper. The naive computation of relative attention requires storing Q × E_r^T as an L × L matrix where entry (i, j) uses E_r[i-j] — a different embedding for each relative offset. Computing this naively requires O(L²D) memory.

Huang et al. observed that this matrix has a specific structure: it's a Toeplitz-like matrix — each diagonal has the same value. Entry (i, j) depends only on (i-j), so all entries on the same diagonal are identical. They exploit this structure with an elegant reshaping trick.

Step 1: Compute Q E_r^T efficiently

Instead of creating L separate embeddings for each query, we use the fact that the relative offset (i-j) ranges from 0 to L-1 (in the causal case). We only need L unique embeddings, stored in E_r of shape [L, d].

Compute the product S_rel = Q E_r^T, which has shape [L, L]:

But we need S_rel[i, j] = q_i^T E_r[i-j], not q_i^T E_r[k]. The index k in our computation corresponds to the relative offset, but we need to rearrange so that column j uses offset (i-j).

Step 2: Skew the matrix

The trick: pad S_rel with one column of zeros on the left, reshape it from [L, L+1] to [L+1, L], then slice off the first row. The resulting [L, L] matrix has exactly the Toeplitz structure we need: entry (i, j) = q_i^T E_r[i-j].

A worked example with 4 positions

Let's trace through with L=4. The raw matrix Q · E_r^T has entry (i,k) = q_i · e_k:

We want entry (i,j) to use e_i-j. So position (0,0) should use e₀, position (1,0) should use e₁, position (2,1) should use e₁, etc. After padding a zero column on the left and reshaping:

Reshape [4, 5] → [5, 4] by reading elements in row-major order and filling a new shape:

Slice off row 0: the resulting [4, 4] matrix has entry (i,j) = e_i-j (reading from row i=0 which is now [e3, 0, e0, e1] — wait, that's not right). Actually, the correct interpretation requires that E_r is indexed in reverse order (e₀ = offset 0, e₁ = offset 1, etc.), and the resulting matrix after slicing rows 1-4 gives the correct Toeplitz structure.

The implementation handles the indexing details automatically through the reshape — the key insight is that the operation is O(1) additional computation (just pointer arithmetic, no new multiply-adds).

Multi-head relative attention

In the multi-head setting, each head has its own relative position embeddings E_r. This lets different heads specialize: one head might learn to attend at beat intervals (every 8 positions), another at bar intervals (every 32 positions), and another at phrase intervals (every 128 positions). The skewing trick is applied independently per head.

python
# Multi-head relative attention with skewing
class RelativeMultiHeadAttn(nn.Module):
    def __init__(self, d_model, n_heads, max_len):
        super().__init__()
        self.d_k = d_model // n_heads
        self.n_heads = n_heads
        self.W_q = nn.Linear(d_model, d_model)
        self.W_k = nn.Linear(d_model, d_model)
        self.W_v = nn.Linear(d_model, d_model)
        self.W_o = nn.Linear(d_model, d_model)
        # Per-head relative position embeddings
        self.E_r = nn.Parameter(
            torch.randn(max_len, self.d_k)
        )
        # Global bias vectors (u, v)
        self.u = nn.Parameter(torch.randn(n_heads, self.d_k))
        self.v = nn.Parameter(torch.randn(n_heads, self.d_k))

    def forward(self, x):
        B, L, D = x.shape
        # Project to Q, K, V  [B, L, H, d_k]
        Q = self.W_q(x).view(B, L, self.n_heads, self.d_k)
        K = self.W_k(x).view(B, L, self.n_heads, self.d_k)
        V = self.W_v(x).view(B, L, self.n_heads, self.d_k)

        # Content-content: (Q + u) @ K.T
        Qu = Q + self.u  # broadcast u over B, L
        S_content = torch.einsum('blhd,bmhd->bhlm', Qu, K)

        # Content-position: (Q + v) @ E_r.T + skew
        Qv = Q + self.v
        E = self.E_r[:L]  # [L, d_k]
        S_pos = torch.einsum('blhd,md->bhlm', Qv, E)
        S_pos = self._skew(S_pos)  # apply skewing trick

        # Combine, mask, softmax
        scores = (S_content + S_pos) / (self.d_k ** 0.5)
        # ... causal mask and softmax ...
        return self.W_o(out)

    def _skew(self, S):
        # S: [B, H, L, L] → skewed [B, H, L, L]
        B, H, L, _ = S.shape
        S = F.pad(S, (1, 0))  # [B, H, L, L+1]
        S = S.reshape(B, H, L+1, L)
        S = S[:, :, 1:, :]  # [B, H, L, L]
        return S

Memory savings

python
def relative_attention_skew(Q, K, V, E_r):
    # Q, K, V: [L, d]
    # E_r: [L, d] — relative position embeddings
    L, d = Q.shape

    # Content-to-content scores (standard attention)
    S_content = Q @ K.T  # [L, L]

    # Content-to-position scores (needs skewing)
    S_rel = Q @ E_r.T  # [L, L] — but wrong indexing!

    # THE SKEWING TRICK:
    # Step 1: Pad with zero column on left
    S_rel = F.pad(S_rel, (1, 0))  # [L, L+1]

    # Step 2: Reshape to shift rows
    S_rel = S_rel.reshape(L + 1, L)  # [L+1, L]

    # Step 3: Slice off first row
    S_rel = S_rel[1:]  # [L, L] — now correctly indexed!

    # Combine and scale
    scores = (S_content + S_rel) / (d ** 0.5)

    # Apply causal mask
    mask = torch.triu(torch.ones(L, L), diagonal=1).bool()
    scores.masked_fill_(mask, float('-inf'))

    # Standard softmax and value aggregation
    weights = F.softmax(scores, dim=-1)
    return weights @ V  # [L, d]

Chapter 5: Generation Results

The Music Transformer was trained on the J.S. Bach Chorales dataset (382 four-part chorales) and the Piano-e-Competition dataset (1,573 virtuoso piano performances). The results demonstrated a clear advantage for relative attention in generating music with long-term structure.

Quantitative results

The negative log-likelihood improvement from 5.52 to 5.36 might seem small, but in autoregressive models, even small NLL improvements translate to noticeably better generation quality. More importantly, the qualitative difference was dramatic.

Training details

The Music Transformer was trained on the Piano-e-Competition dataset — 1,573 performances by virtuoso pianists, totaling ~172 hours of music. Each performance was converted to the MIDI event representation, giving sequences of 1,000-3,000 events. Training used:

The model is relatively small by modern standards (~10M parameters), but this was sufficient because the vocabulary is tiny (388 events) and the task is well-structured. Larger models didn't significantly improve quality — the bottleneck was training data, not model capacity.

Evaluation methodology

Evaluating music generation is notoriously difficult — there's no BLEU score for music. The paper used three evaluation methods:

The human evaluation was critical. NLL measures density estimation quality, but a model with good NLL might generate boring, repetitive music (the "safe average"). The listening study confirmed that the Music Transformer's outputs were not just statistically good but actually musical — with structure, development, and emotional arc.

Qualitative analysis

Human evaluators consistently preferred the Music Transformer's outputs. The key differences:

Self-attention visualized on real music

The paper includes attention visualizations on real performances from the Piano-e-Competition dataset. These visualizations confirm that the model learns musically meaningful patterns:

The attention patterns reveal something fascinating: different heads specialize for different musical functions. Some heads learn a strong diagonal pattern (attending to events at regular beat intervals — discovering the meter without being told). Other heads attend to events with similar pitch content (tracking the melody). Still others attend to the most recent events (local context for harmony). This emergent specialization is possible precisely because relative position encodings allow the model to express "attend to events N steps back" as a learned pattern.

python
# The Music Transformer architecture
class MusicTransformer(nn.Module):
    def __init__(self, vocab=388, d_model=256,
                 n_heads=8, n_layers=6, max_len=2048):
        super().__init__()
        self.embed = nn.Embedding(vocab, d_model)
        # Relative position embeddings (shared across layers)
        self.E_r = nn.Embedding(max_len, d_model // n_heads)
        self.layers = nn.ModuleList([
            RelativeAttentionBlock(d_model, n_heads, max_len)
            for _ in range(n_layers)
        ])
        self.ln = nn.LayerNorm(d_model)
        self.head = nn.Linear(d_model, vocab)

    def forward(self, x):
        # x: [B, L] — event token ids
        h = self.embed(x)       # [B, L, D]
        # NO absolute position added!
        for layer in self.layers:
            h = layer(h, self.E_r)
        h = self.ln(h)
        return self.head(h)    # [B, L, 388]

Chapter 6: Music Structure Explorer

Let's explore how the Music Transformer captures musical structure with an interactive simulation. This is the showcase — you'll see how relative attention enables the model to generate coherent patterns.

Why this matters for musical structure

The three levels of musical structure that relative attention captures:

The LSTM can handle local structure well. The Transformer with absolute position handles local and some phrase-level structure. The Music Transformer with relative attention handles all three levels — and this is what makes its generated music sound structured rather than wandering.

Why relative attention creates repetition

Here's the mechanism: when the model learns that "a NOTE_ON event should attend strongly to the event 32 steps back" (via the Content-Position term in the attention score), it creates a natural copying mechanism. If the event 32 steps back was a C, the model is biased toward producing a C again. If the attention pattern says "attend at offset -32 and offset -64," the model creates a repeating pattern with period 32. This is exactly how musical phrases repeat — and the model discovers it without being told anything about musical structure.

What each model produces over time

The best way to understand the Music Transformer's advantage is to compare outputs at different time horizons:

The difference at the 60-120 second mark is the most telling. LSTMs have completely lost the thread. Absolute-position Transformers produce locally coherent music but without structural backbone. Only the Music Transformer with relative attention generates music that sounds like it has a plan — because relative attention enables the "copy from N steps back" behavior that creates repetition and development.

Temperature and sampling

During generation, the model's output logits are divided by a temperature parameter before softmax. Lower temperature (0.7-0.9) makes the model more deterministic, favoring the most likely events — producing safer, more conventional music. Higher temperature (1.0-1.3) adds randomness, producing more creative but potentially less coherent output. The paper found that temperature around 1.0 gave the best balance of coherence and creativity for piano performances.

python
# Generating music with the Music Transformer
def generate(model, primer, length, temperature=1.0):
    # primer: initial MIDI events (e.g., first bar)
    generated = list(primer)

    for _ in range(length):
        x = torch.tensor([generated])  # [1, T]
        logits = model(x)[0, -1]     # [388] — last position

        # Temperature scaling
        probs = F.softmax(logits / temperature, dim=-1)

        # Sample from distribution
        next_event = torch.multinomial(probs, 1).item()
        generated.append(next_event)

    return generated  # List of MIDI event tokens

Chapter 7: Connections — From Music to Modern Audio AI

The Music Transformer's contributions extend far beyond music generation. Relative position representations became a standard component of modern Transformers, and music generation evolved into a broader field of audio AI.

The lineage of relative position

Evolution of music AI

What changed

The biggest shift: moving from symbolic (MIDI) to raw audio. The Music Transformer generates MIDI events — a score, like sheet music. To hear it, you need a synthesizer. Modern models like MusicLM and MusicGen generate raw audio directly — including timbre, dynamics, and even vocals. This required new tokenization schemes (VQ-VAE codebooks instead of MIDI events) and much more compute.

The tokenization revolution

The progression in audio tokenization mirrors the progression from character-level to subword-level models in NLP:

The key innovation: neural audio codecs like SoundStream (Google) and EnCodec (Meta) that compress audio into discrete tokens at ~50 tokens/second — compact enough for a Transformer to model. MusicLM and MusicGen both build on these codecs, using the Transformer to model sequences of audio tokens rather than MIDI events.

Limitations and what came after

The Music Transformer had notable limitations:

python
# The progression of music AI architectures

# 2019: Music Transformer
# Input: MIDI events (388 vocab, ~2000 tokens/piece)
# Model: Transformer decoder + relative attention
# Output: MIDI events → render with synthesizer

# 2020: Jukebox
# Input: raw audio → VQ-VAE → discrete codes
# Model: Sparse Transformer (3 levels of hierarchy)
# Output: discrete codes → VQ-VAE decoder → raw audio

# 2023: MusicGen
# Input: text description + (optional) melody
# Model: Transformer decoder over interleaved codebooks
# Output: EnCodec tokens → EnCodec decoder → raw audio

# Key: the Transformer stayed constant; only the
# tokenization evolved (MIDI → VQ-VAE → EnCodec)

But the core architecture remained: the Transformer with some form of relative position encoding. The Music Transformer proved that attention could capture long-range musical structure. Everything since has been about scaling that insight to richer audio representations.

The RoPE connection

The most widely used relative position encoding today is Rotary Position Embedding (RoPE, Su et al., 2021). RoPE encodes relative position by rotating the query and key vectors in the complex plane. The dot product q^Tk naturally decomposes into a content component and a position component that depends only on the offset (i-j):

Where R_t is a rotation matrix that rotates by angle t×θ. The key insight: the rotation makes the dot product depend only on the difference (i-j), achieving the same relative position effect as the Music Transformer but more elegantly — no skewing trick needed, no separate position embedding table.

python
# RoPE: the modern descendant of relative attention
def apply_rope(q, k, positions, d_model, theta=10000):
    # q, k: [B, T, d]
    # Compute rotation frequencies
    freqs = 1.0 / (theta ** (torch.arange(0, d_model, 2) / d_model))
    # Build rotation angles: [T, d/2]
    angles = positions[:, None] * freqs[None, :]
    # Apply rotation (complex multiply)
    cos_a, sin_a = torch.cos(angles), torch.sin(angles)
    # Rotate pairs of dimensions
    q1, q2 = q[..., ::2], q[..., 1::2]
    k1, k2 = k[..., ::2], k[..., 1::2]
    q_rot = torch.cat([q1*cos_a - q2*sin_a, q1*sin_a + q2*cos_a], dim=-1)
    k_rot = torch.cat([k1*cos_a - k2*sin_a, k1*sin_a + k2*cos_a], dim=-1)
    return q_rot, k_rot
    # Now q_rot @ k_rot.T depends only on (i-j)!

What remains unsolved

Despite the enormous progress since 2019, several challenges from the Music Transformer era remain open:

The Music Transformer showed that relative attention could capture phrase-level repetition (~30 seconds). Getting to movement-level structure (~5 minutes) and piece-level form (~20 minutes) remains an open challenge. The attention mechanism can theoretically handle these timescales if the context window is long enough, but training data with clear large-scale structure is scarce, and evaluation is subjective.

Perhaps the most important open question is: does generating coherent musical form require understanding music (in some computational sense), or can it emerge purely from pattern matching on enough training data? The Music Transformer suggests the latter — its relative attention heads discovered beats, phrases, and repetition structure purely from statistical patterns in MIDI data, with no explicit musical knowledge. If this scaling hypothesis holds, then sufficiently large models trained on enough music data might eventually generate pieces with genuine formal coherence. Whether this constitutes "understanding" music is a philosophical question the paper wisely leaves open.

python
# Summary: Music Transformer contributions

# 1. Relative attention for sequences
#    - Replaces absolute PE with relative offset embeddings
#    - Attention(i,j) depends on (i-j), not on i and j separately
#    - Enables pattern recognition invariant to position

# 2. The skewing trick for efficient computation
#    - Reduces memory from O(L²D) to O(LD)
#    - Zero additional compute (just pad + reshape)
#    - Makes relative attention practical for long sequences

# 3. Event-based MIDI representation
#    - 388-token vocabulary captures full piano performance
#    - 1000x compression vs piano roll
#    - Includes dynamics (velocity) and timing nuance

# 4. Proof that attention captures musical structure
#    - Heads discover beats, phrases, harmony automatically
#    - Long-range coherence up to 2 minutes
#    - Preferred by human listeners over LSTM 70% of the time

# 5. Influence on all subsequent position encodings
#    - Transformer-XL (2019): relative + segment recurrence
#    - RoPE (2021): rotary encoding, used in Llama/Gemma
#    - ALiBi (2022): linear bias, no learned params
#    - Every modern LLM uses relative position encoding

# The Music Transformer was a small model (10M params)
# on a small dataset (172 hours of piano).
# But its ideas — relative attention, the skewing trick,
# event-based tokenization — influenced every model that
# came after. Sometimes the most important papers are not
# the biggest, but the ones that ask the right question.