MIDI-LLM: Adapting Large Language Models
for Text-to-MIDI Music Generation

Chapter 1: The Key Insight

Here is the bet of the entire paper, in one sentence: don't build a music model — adapt a language model. Take a pretrained text LLM (Llama 3.2, 1B parameters), teach it that MIDI is just another "language" it can read and write, and let it inherit everything the LLM already knows.

Why is that a good bet? Two reasons the authors lean on:

What the whole recipe is, in one breath

MIDI-LLM is: take Llama 3.2 1B → widen its embedding/output layers to hold 55K new MIDI tokens alongside the 128K text tokens (Ch 3) → train in two stages, first on lots of music-adjacent text and standalone MIDI, then on MIDI paired with text captions (Ch 5) → at inference, feed a text prompt as a prefix and let the model autoregressively emit MIDI tokens, three tokens per note (Ch 4) → serve it with vLLM at FP8 for speed (Ch 6). Each piece is built up from scratch below, with the actual tensor shapes.

Chapter 2: MIDI Tokenization — Notes as Tokens

Before an LLM can "speak" music, the music has to become a sequence of discrete tokens — the same way text becomes a sequence of subword IDs. There are several competing schemes (MIDI-like, REMI, ABC notation). MIDI-LLM adopts the arrival-time MIDI-like tokenization from the Anticipatory Music Transformer (AMT, Thickstun et al. 2024).

The reason for that choice is itself instructive: REMI and ABC tokenizations require the music to be beat-synchronized (quantized to a metrical grid). AMT's arrival-time scheme doesn't — it measures time in plain seconds, so it works on any MIDI, including expressive performances that don't sit on a clean grid. More flexibility, fewer prerequisites on the data.

The three token types, with their ranges

Why joint instrument-pitch instead of two separate tokens? It shortens the sequence (2 tokens of info packed into 1) and ties the two facts that always co-occur — a "piano C4" is one concept, not an independent instrument and an independent pitch. The cost is a big vocabulary: 129 × 128 = 16,512 instrument-pitch tokens alone.

Counting the vocabulary — a worked example

Let's count exactly where AMT's 55K music tokens come from, because the number reappears in the next chapter:

That last doubling is worth a beat. AMT keeps a second identical set of tokens called anticipated tokens. These represent future notes that are handed to the model as conditions — they let the model do infilling (fill a gap in the middle of a piece given the notes that come after it). So the vocabulary is doubled: one copy for notes the model generates, one copy for notes it's told about in advance.

Click "Add a note" to drop a note on the piano roll (top). Each note instantly emits three tokens into the sequence below — arrival time, duration, instrument-pitch — color-coded. Watch the token stream grow at exactly 3 tokens per note. This regular rhythm is what the LLM learns to continue.

Chapter 3: Vocabulary Expansion

Now the crux. The LLM speaks 128K text tokens; AMT defines 55K music tokens. How do you get one model to read and write both in a single sequence?

The naive option — and the one the authors explicitly reject — is to serialize MIDI as text: write each note out as the literal string <onset 10.2s> <duration 120ms> <piano, C4> and feed it to Llama's normal text tokenizer. Why is that bad? Because Llama's tokenizer would shatter that string into many subword pieces — "onset", "10", ".", "2", "s", and so on — bloating one note into a dozen-plus tokens. Longer sequences mean more compute per note and a harder learning problem.

The one equation in the paper, decoded

The token-embedding matrix of the LLM is E_LLM. We stack a new block of music embeddings on top of it:

Symbol by symbol, with the plain meaning:

• E_LLM — the original text embedding table. Shape |V_LLM| × D = 128K rows, each a D-dimensional vector. Row i is "what text token i means" to the model. We copy it over unchanged.
• E_AMT — a new table for the 55K music tokens, shape |V_AMT| × D. Initialized randomly — the model knows nothing about MIDI tokens yet; training will fill these in.
• D — the hidden dimension (the width of every vector flowing through the transformer). For Llama 3.2 1B, D = 2048. This must match — that's the whole point of "preserving the parameter structure": the new music vectors live in the same space as the text vectors.
• The stacking [\cdot] — vertically concatenate the two tables into one taller table of 128K + 55K = 183K rows. The transpose notation is just bookkeeping for "stack along the vocabulary axis."

The teal block is the inherited 128K-row text embedding table (copied, already meaningful). Click "Expand" to stack the 55K-row music table (orange) on top — randomly initialized noise that training must turn into real music-token vectors. The width D = 2048 is shared, so music vectors live in the same space as words.

What also changes — the output head

An LLM ties (or mirrors) its output projection to the same vocabulary: to predict the next token it produces a logit for every vocabulary entry. So expanding the input embeddings means the output layer also grows from 128K to 183K logits per position. The model can now emit music tokens, not just read them. During generation it will simply pick the highest-probability token among all 183K — sometimes a word, usually (once it's writing MIDI) one of the three note-token types.

# Vocabulary expansion: the core of MIDI-LLM, ~5 lines
import torch
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B")
V_text = model.config.vocab_size      # 128256
V_amt  = 55000                      # AMT music tokens (incl. anticipated)

# resize_token_embeddings copies old rows, random-inits the new ones,
# and grows BOTH the input embedding AND the tied output head
model.resize_token_embeddings(V_text + V_amt)

# shapes after expansion (D = 2048 for Llama 3.2 1B)
print(model.get_input_embeddings().weight.shape)   # [183256, 2048]
print(model.lm_head.weight.shape)                  # [183256, 2048]
# Everything else (attention, MLP, norms) is UNCHANGED -> vLLM works.

Chapter 4: Generation (the showcase)

Now we can watch the model actually work. At inference time, the recipe is exactly like prompting a chatbot — except the "answer" is music.

1. The user's text prompt (genre, mood, instrumentation, tempo, key, chords) is tokenized with the text tokens and laid down as a prefix — up to 256 tokens.
2. The model then autoregressively emits music tokens, one at a time, each conditioned on everything before it: the prompt and all notes generated so far.
3. Tokens arrive in the fixed 3-token rhythm: arrival, duration, instrument-pitch, arrival, duration, instrument-pitch, ... Every three tokens snap together into one playable note.

The objective being unrolled here is the ordinary autoregressive factorization — the same one a text LLM uses, now over a sequence that begins with words and continues with music:

where s_1..k are the text-prompt tokens (given, not predicted) and s_k+1..T are the music tokens the model generates. There is no special "music decoder" — it is the same transformer predicting the next token, every step.

Sampling: nucleus (top-p) at p = 0.98

At each step the model gives a probability over all 183K tokens. You don't take the single most likely token (that produces repetitive, lifeless music) — you sample. MIDI-LLM uses nucleus sampling with p = 0.98: keep the smallest set of top tokens whose probabilities sum to 0.98, renormalize, and draw from that set. High p keeps diversity (musical surprise) while still cutting off the long tail of nonsense tokens — "to balance musical diversity and coherence," in the paper's words.

The text prompt is fixed as a prefix (gray). Hit "Sample next token" to emit one token at a time and watch the 3-token cycle build a note onto the piano roll. The bar chart on the right is the model's distribution over candidate tokens — the shaded region is the nucleus (top-p) it samples from. Lower top-p to make it greedier/more repetitive; raise it for more variety. "Auto-generate" runs the loop.

# Text-to-MIDI generation is just LLM .generate() with a prompt prefix
prompt = "An upbeat jazz piece with a violin solo, tempo 140, key C major."
ids = text_tokenizer(prompt, return_tensors="pt").input_ids   # [1, k]

out = model.generate(
    ids,
    max_new_tokens=2048,       # ~30 s of music; multiple of 3 -> whole notes
    do_sample=True,
    top_p=0.98,              # nucleus sampling: diversity vs coherence
)
music_ids = out[0, ids.shape[1]:]    # strip the text prefix

# every 3 music tokens = one note: (arrival, duration, instrument-pitch)
notes = [amt_detokenize(music_ids[i:i+3]) for i in range(0, len(music_ids), 3)]
write_midi(notes, "out.mid")              # editable MIDI file!

Chapter 5: The Two-Stage Training Recipe

Random music embeddings (Ch 3) don't know anything yet. Training fills them in — and the paper does it in two deliberate stages, both using the same plain next-token cross-entropy loss. The split mirrors a common LLM specialization pattern: first broadly adapt to the domain, then sharpen on the exact target task.

Stage 1 — Continued pretraining (~3 B tokens)

Two goals, the paper states them explicitly:

• Surface latent musical knowledge. Train on music-adjacent text (a subset of MusicPile: music articles, GPT-4-synthesized music knowledge, music QA in ABC notation, common-knowledge QA). This "wakes up" whatever the LLM already half-knew about music from its original pretraining.
• Learn MIDI syntax. Train on standalone MIDIs (GigaMIDI, minus the pieces reserved for stage 2) so the random music embeddings learn the structure of AMT tokens — what sequences of arrival/duration/instrument-pitch tokens form coherent music. No text pairing yet; this is pure "learn the language of MIDI."

Stage 2 — Supervised finetuning (~1.7 B tokens, 5.1 B after augmentation)

Now teach the actual translation. Each training example is a text caption as prefix (from MidiCaps — genre, mood, instrumentation, tempo, tonality, chords) followed by the AMT tokens of the matching MIDI (from Lakh MIDI). The model learns: given this description, produce these notes.

Click each box. Stage 1 (continued pretraining) feeds music-text + standalone MIDI to learn vocabulary and MIDI syntax. Stage 2 (finetuning) feeds paired caption→MIDI to learn the text-to-music translation. The arrow shows the model carrying forward what it learned.

The training facts, for the record

Chapter 6: Fast Inference — Cashing the Ecosystem Dividend

This is where "preserve the LLM's parameter structure" pays off in wall-clock seconds. Because MIDI-LLM is a standard Llama decoder, it drops straight into vLLM — the same serving library that accelerates text LLMs — and inherits three optimizations with zero custom engineering.

Reading the speed numbers — the real-time factor (RTF)

The headline metric is RTF = (duration of generated music) ÷ (wall-clock time to generate it). RTF > 1 means faster than real time — you generate music quicker than it would take to play. Higher is better. The worked comparison, on a single L40S GPU for a 2K-token clip:

Walk the arithmetic for FP8, bsz=1: it produced 32.6 s of music in 8.1 s of compute → RTF = 32.6 / 8.1 = 4.02. Text2midi managed RTF 0.56 — slower than real time. So MIDI-LLM is ~7× better on RTF despite being 5× bigger, and batching (bsz=4) pushes it to 14× real time. FP8 buys the last ~20% on top of BF16.

Bars are RTF (generated seconds per wall-clock second). The dashed line at RTF = 1 is "real time" — Text2midi sits below it at bsz=1. Toggle batch size to watch MIDI-LLM's advantage explode, because a standard decoder batches efficiently in vLLM while a custom architecture does not.

Chapter 7: Experiments & Results

Both quality metrics need audio input, but MIDI-LLM outputs MIDI — so the authors synthesize each generated MIDI into audio with FluidSynth first, then measure. Two automatic metrics, both standard in text-to-music evaluation:

The headline table (MidiCaps test subset, 896 samples)

Read it carefully. MIDI-LLM wins decisively on both axes — quality (FAD drops by ~4.7×) and text control (CLAP up 3.4 points). And FP8 quantization barely dents quality (FAD 0.173 → 0.216, CLAP 22.1 → 21.8) while buying the speedup from Ch 6. That tiny gap is the engineering sweet spot: near-free deployment savings.

Toggle between the two metrics. Notice the FP8 bar barely differs from BF16 — quantization is nearly free in quality — while both crush the Text2midi baseline. The direction of "better" flips between the metrics, which the axis label reminds you of.

Chapter 8: The Negative Findings

Good papers report what didn't work, and MIDI-LLM offers two honest negative results that are arguably more interesting than the wins — because they puncture assumptions the design was built on.

Why might this happen? Think about the information balance. For pure text-to-MIDI generation, the prompt is the only signal — the model must lean on it. For infilling, the surrounding notes are an enormously strong, low-entropy constraint (music is locally very predictable), so the cheap, reliable strategy is "match the neighbors" and treat the text as a minor nudge. The model learned the path of least loss.

The authors are appropriately cautious — they flag confounders (the two corpora differ in curation quality, not just topic). But the implication is provocative: maybe the LLM's musical world-knowledge is already baked in from its original pretraining, and the continued-pretraining text mostly serves to keep the language abilities from degrading while the model learns MIDI syntax — not to "teach music." Both findings are explicitly flagged as open directions for future work.

Drag the slider for how much surrounding MIDI context the model sees around a gap. With little context (pure generation, left) the text prompt dominates the decision; as context grows (infilling, right) the surrounding notes take over and the prompt's influence collapses — the paper's negative finding, visualized as a tug-of-war.

Chapter 9: Connections & Cheat Sheet

MIDI-LLM sits at the meeting point of two lineages: the symbolic-music transformers (Music Transformer → AMT → MIDI-LLM) and the "adapt a general LLM to a new modality by expanding its vocabulary" pattern that recurs across speech, code, and now music.

The cheat sheet — every key fact

The one-paragraph summary you could give on a whiteboard

MIDI-LLM turns a text LLM into a text-to-MIDI generator by expanding its vocabulary: stack 55K AMT music tokens (3 per note — arrival, duration, instrument-pitch) onto Llama 3.2 1B's 128K text tokens, sharing the same 2048-d embedding space. Train in two stages with plain next-token loss — first continued pretraining on music-adjacent text + standalone MIDI to learn MIDI syntax, then supervised finetuning on caption→MIDI pairs (tripled via AMT infilling augmentation). At inference the text prompt is simply the prefix of one shared sequence the model continues with music tokens (nucleus sampling, p=0.98). Because the model is a vanilla Llama decoder, it serves on vLLM with paged attention and FP8, hitting real-time factor ~4 — versus 0.56 for the custom Text2midi baseline — while scoring far better on FAD (quality) and CLAP (text control). Two honest negatives: infilling mostly ignores the text, and music-adjacent pretraining text didn't measurably help.