Give a model a melody and it composes a harmonizing accompaniment — by interleaving the controls a few seconds early into one autoregressive sequence, so an ordinary GPT-style transformer can plan ahead without any encoder, cross-attention, or long-range trickery.
You are handed a melody — a singer's line, say — and asked to write a piano part that fits under it. A bass, some chords, a countermelody. This is the everyday act of accompaniment, and it is the motivating problem of this paper.
Phrase it as a machine-learning task. The melody is a fixed set of musical events the user gives you: call them the controls. The accompaniment is the set of events you must generate: the events. You want a model that can sample a plausible accompaniment given the melody — formally, sample from p(events | controls).
The obvious move is Seq2Seq: write down the whole melody first, then the whole accompaniment after it — u1, ..., uK, e1, ..., eN. Now when the model is generating the accompaniment note that belongs at, say, second 90 of the song, the melody note it most needs to look at — the one also at second 90 — is hundreds of tokens behind it in the sequence.
The model has to learn an artificial long-range dependency: "to write the note here, go reach back 600 tokens to find the melody note from the same moment." That is exactly the kind of long-range bookkeeping transformers are bad at and burn enormous capacity trying to learn.
The paper's premise is the opposite: the context you actually need to predict the next note is local — the recent accompaniment you just wrote (causal history) plus the melody notes happening right around now (a peek into the near future). So why not physically place each melody note next to the accompaniment notes it belongs with?
Lakh MIDI music runs at an average of 68 tokens per second. A transformer context of M = 1024 tokens therefore spans only about 15 seconds of music. If the melody note you need is 90 seconds away, it is not even in the context window. No amount of long-range modeling helps — the information is simply gone.
| Quantity | Value | Why it matters |
|---|---|---|
| Avg. token rate (Lakh MIDI) | 68 tok/s | How fast the sequence fills up |
| Context length M | 1024 tokens | ≈ 15 seconds of music fits at once |
| Seq2Seq melody offset | up to thousands of tokens | Melody note can fall outside the window entirely |
| Goal: control on time t appears within | ~M tokens of events near t | Locality is the whole design target |
That last row is the design target in one line: put each control close, in the sequence, to the events it constrains. Everything in this paper is the careful machinery needed to do that and still have a sequence you can sample from one token at a time.
Here is the whole idea in a sentence. Take the controls (melody) and the events (accompaniment) and weave them into one sequence, placing each control a fixed interval δ seconds before the moment it controls. The model reads this woven sequence left to right and predicts the next token, just like any language model.
Read the paper's own framing through a control-theory lens. Predicting the next event from your local history is a filtering (causal) estimate — you only use the past. Letting near-future controls into the context adds a smoothing (bidirectional) estimate — you peek slightly ahead. Anticipation gives you a filtering estimate over events fused with a smoothing estimate over controls, in one ordinary causal model.
And δ is the dial. Too small and the model is blind to upcoming controls (no lead time to plan). Too large and you destroy locality — the control drifts so far ahead that it leaves the context window, and you are back to Seq2Seq. In fact, the paper notes that δ = ∞ recovers Seq2Seq exactly. Anticipation is the interpolation between "everything local" and "all controls up front."
"Just insert the control δ seconds early" sounds simple — and it would be, if you knew the whole sequence in advance. But during generation you are building the sequence one token at a time and you do not yet know the events you have not generated. The rule for where to insert a control has to be computable from the tokens you have already produced. Getting that rule right is the entire technical core of the paper, and it hinges on a concept called a stopping time (Chapter 4).
Before we can interleave anything, we need a precise mathematical object for "music." The paper uses a marked temporal point process: a probability distribution over sparse events that occur at points in continuous time, where each event carries a label (a "mark").
An event is a pair ei = (ti, mi). Here ti ∈ ℝ+ is when the event happens (time-ordered: ti ≤ tj for i < j), and mi is the mark — what happens — drawn from a finite vocabulary V.
The paper restricts marks to notes, each described by a pitch p, an instrument class k, and a duration d. Following MIDI: pitch p ∈ {0,...,127} on the 12-tone scale (p = 60 is middle C, 261 Hz); instrument k ∈ {0,...,128} (code 128 = drums); duration in seconds quantized to 10 ms steps, capped at 10 s.
Pitch and instrument are bundled into a single number n = 128k + p. So a mark is mi = (di, ni) drawn from a vocabulary of 17,512 marks, and a piece of music is just these notes scattered in time.
Given controls u1:K from the user, we can control generation of events e1:N if we can sample from p(e1:N | u1:K). The paper focuses on infilling control, where the controls share a vocabulary with the events: the user fixes some notes, and the model fills in a complete piece that contains those notes. Accompaniment is the special case where the fixed notes are an entire melody line. (We return to infilling in Chapter 8.)
A transformer eats tokens, not continuous-time events. So we must serialize each event into integers. The paper's key encoding choice — the one that makes anticipation possible — is arrival-time tokenization.
First, quantize time. Arrival times use a vocabulary of 10,000 values, in 10 ms steps, up to 100 seconds (times are measured relative to the start of the context, so this is plenty). Then each event becomes exactly three tokens:
So event i is the triplet (ti, di, ni). The combined vocabulary is 27,512 tokens: 10,000 times + 1,000 durations + 16,512 instrument-pitch pairs. A whole piece is the triplets laid end to end: x1:3N.
The more common encoding (REMI, OctupleMIDI, and the lineage of Music Transformer) is interarrival-time: instead of storing each note's absolute time, you store the gap Δi = t'i+1 − t'i since the previous token. Events split into onset/offset tokens separated by these gap tokens.
This is context-sensitive: a token's actual time is determined by where it sits in the sequence, because time is accumulated from all the gaps before it. Move a token and every subsequent timestamp shifts. You therefore cannot freely reorder — which kills anticipation before it starts.
Drag the three note dots along the timeline. Watch how arrival-time tokens (top) carry absolute times that don't care about order, while interarrival tokens (bottom) recompute every gap when the spacing changes.
Now the subtle part. We want to place a control on time s "near the events at time s − δ." The naive way is sort order: pretend the control is an event at time s − δ and merge it in by timestamp — slot it between event tj and tj+1 where tj ≤ s − δ < tj+1.
This is fine if you already have the whole piece. But during generation you are emitting tokens one at a time and you only ever see a prefix of the sequence. To slot the control between tj and tj+1, you need to know both the event before it and the event after it. But the event after it hasn't been generated yet.
The remedy is to define insertion using only the prefix you have already seen. That is exactly what a stopping time is. Informally: a random index τ is a stopping time if you can decide "did τ = i happen?" using only the tokens up to index i — no peeking ahead.
Events at t1=1, t2=3, t3=5; one control at s1=7; interval δ=5, so s1−δ=2.
The two orders differ by one position — and that one position is the difference between a model you can run and one you can't.
We can now state the rule precisely. Given δ > 0, build a1:N+K = interleaveδ(e1:N, u1:K). Define t0 = s0 = −∞ for bookkeeping. Then event ej and control uk land at indices:
Unpack Equation (4), the one that matters: control uk comes after the first k−1 controls (the k term) and after the first event ej whose time reaches sk − δ (the argmin term). "First event to reach" = a first-hit time = a stopping time. That is the whole trick, made formal.
This is the paper's Figure 1, made interactive. Drag the teal events and warm controls along the timeline; turn the δ slider. The bottom row shows the serialized sequence the model would actually see, built by the rule above. Toggle to compare against the broken sort order and against Seq2Seq. Watch each control snap to the position right after the first event that reaches s − δ.
The rule has a failure mode, and the paper is honest about it. Anticipation promises a control will land somewhere near the events at time s − δ — but it guarantees nothing about how many seconds early. If the music is sparse (big silences between notes), a control can even land after the time it was supposed to control.
Events at t1=1, t2=2, t3=5 (note the 3-second gap), control s1=4.5, δ=2, so s1−δ=2.5. The first event reaching 2.5 is e3 at t=5. So u1 lands after e3: the sequence is (e1, e2, e3, u1).
The culprit is the gap. Define Δmax = the largest gap between adjacent events; the sequence is Δmax-dense. The guarantee is: a control on time t appears at or before time t − δ + Δmax. If Δmax > δ, the bound goes positive and you risk late controls. In the example Δmax=3 > 2=δ — exactly the danger zone.
Insert special REST events to keep the sequence dense. Pick a target density Δ*; whenever a gap exceeds it, drop REST markers to fill it. Formally, if nΔ* < ti+1 − ti ≤ (n+1)Δ*, insert n RESTs at times ti+Δ*, ti+2Δ*, .... All paper models use Δ* = 1 second.
Continuing the example with Δ*=1: RESTs at times 3 and 4 turn the events into e1,e2,REST(3),REST(4),e3, and now u1 tucks neatly between the time-3 and time-4 RESTs: (e1,e2,REST,u1,REST,e3). The control is anticipated again, on time.
Widen the gap with the slider and watch the control fall behind its own time (red). Lower Δ* to insert RESTs and pull the control back to where it belongs.
Here is the payoff of all that careful ordering: once you have the anticipatory sequence a1:N+K, nothing exotic happens. You tokenize it with the arrival-time encoding and train an ordinary decoder-only transformer by next-token maximum likelihood:
No encoder, no cross-attention, no special architecture. The intelligence is all in the ordering of the sequence, not in the network. That is the elegance: anticipation is a data-layout idea, so you get to reuse the entire mature stack of causal transformer training (here, Stanford's Levanter library).
You can either predict only the events (zero out the loss on control tokens → a purely conditional model) or predict everything (a joint model over events and controls). The paper predicts everything, for a concrete reason: it maximizes the number of supervised predictions per example — all M−1 of them — and for infilling, predicting events and predicting controls are similar enough tasks that learning one reinforces the other.
Generation is just standard autoregressive sampling with one extra check. Track a clock at the most recent event's time t. Before sampling each new event, ask: are there user controls with time ≤ t + δ? If so, splice them in now (anticipate them). Then sample the next event from the model and advance. Repeat until SEP.
Three logit masks enforce validity: arrival times must be non-decreasing (monotonicity); time→duration→note token order must hold (proper triplets); and the model is forbidden from emitting control tokens itself (the user owns those). The while time(controls[k]) ≤ t+delta test is precisely the stopping-time condition from Chapter 5, now doing its job at inference.
So far controls were a separate stream (a melody you hand in). Infilling is the elegant special case where the controls are a subset of the events themselves. The user fixes some notes of a piece and asks the model to complete the rest, in a way that contains the fixed notes. Accompaniment is infilling where the fixed subset is an entire instrument line.
Because triplets are context-free (Chapter 3), the infilling combined sequence a1:N is literally just the original event sequence e1:N re-ordered — some notes pulled out, re-tagged as controls via φ, and anticipated δ seconds early. To recover the played music you undo the tags and sort by time:
Read it: take the generated events e', take the control notes mapped back to the event vocabulary φ−1(u), union them, sort on arrival time. Out comes a coherent piece that contains every user note.
In training there is no built-in split between events and controls — the user could fix any subset at inference. So the paper imposes a prior over control subsets that mimics how people actually use infilling: a mixture of (a) random spans of time (fill this gap), (b) random subsets of instruments (write a bass under this), and (c) uniformly random individual events (sprinkle constraints). Training on this mixture teaches the model to honor whatever subset a user later fixes.
If you fix an instrument the model rarely saw paired with the rest (a weird control subset far outside the training prior), the accompaniment quality degrades — the model has no learned conditional for that situation. The prior is doing heavy lifting: it defines the space of infilling tasks the model is fluent in.
The models are decoder-only causal transformers, context M=1024, trained on the Lakh MIDI dataset: 178,561 files → ~663 million events → ~1.99 billion tokens, ~8,943 hours of music. Three GPT-2-style scales: Small (128M), Medium (360M), Large (780M). Anticipation interval δ=5 seconds, density Δ*=1.
The central empirical claim is in the loss table. Two findings drive it: arrival-time encoding trains to lower perplexity than interarrival-time (validating Chapter 3), and anticipatory training costs almost nothing versus a plain autoregressive baseline.
Lower is better. Toggle scale; the "AM" bars are anticipatory models, the others autoregressive baselines. Notice the AM tax shrinks to nothing with longer training, and how scale + steps both help.
Automatic loss is necessary but not sufficient; music is judged by ears. The paper ran a human listening study on 20-second accompaniment clips, mixing model-generated accompaniments with human-composed ones. The striking finding: listeners rated the anticipatory model's accompaniments with similar musicality to human-composed accompaniments over those clips. The async-control problem, solved well enough to be confused with a human arranger.
The Anticipatory Music Transformer is a lesson that generalizes well beyond music: when you want to condition a causal generative model on an asynchronous, correlated side-channel, you may not need a fancier architecture — you may just need a smarter ordering of one sequence, anchored to stopping times.
| Idea | Rule / Equation | What each symbol means |
|---|---|---|
| Marked point process | ei = (ti, mi) | t = arrival time; m = mark (note) from vocab V |
| Arrival-time tokenization | (ti, di, ni) — 3 tokens | absolute time → triplets are context-free / re-orderable |
| Conditional goal | p(e1:N | u1:K) | e = events to generate; u = user controls |
| Control placement | τuk = k + argminj{tj ≥ sk−δ} | insert after first event reaching sk−δ (a stopping time) |
| Anticipation interval | δ > 0 (δ=∞ ⇒ Seq2Seq) | seconds of lead time; trades planning vs locality |
| Density / RESTs | insert REST if gap > Δ* | guarantee control ≤ time t−δ+Δmax; Δ*=1s |
| Sampling check | while time(uk) ≤ t+δ: emit uk | the stopping-time test, run at inference (Algorithm 1) |
| Infilling | uk = φ(e), recover by sort | controls = re-tagged subset of events; whole seq is a re-ordering |
Model music as a marked temporal point process and tokenize each note as an absolute-time triplet (time, duration, note) — which makes the sequence freely re-orderable. To condition on a melody, weave its notes into the same sequence δ seconds early, inserting each control right after the first generated event whose time reaches s−δ. That insertion index is a stopping time, so the decision uses only the prefix — which is exactly why you can still sample left-to-right (the naive "sort order" can't, because it needs the future). Pad silences with RESTs so controls never fall behind. Then train and sample a perfectly ordinary causal transformer. The result matches unconditional quality "for free" and produces accompaniments humans rate as comparable to human-composed ones.