Music Arena: Live Evaluation
for Text-to-Music

Chapter 1: The Key Insight

The fix borrows a recipe that already worked for chatbots (Chatbot Arena), text-to-speech, image generation, and code assistants. The paper calls it live evaluation, and it has one defining move:

The structure is always the same three steps:

Because every model is scored under the same protocol and the same annotator population, the scores are finally comparable across systems. Because the data flows from genuine usage, it's realistic. Because users keep coming, it's renewable. That single design hits all four wounds from Chapter 0 at once.

But music is not a chatbot

Music Arena's real contribution is recognizing that you can't just copy Chatbot Arena. Music has three properties that text doesn't, and each forces a music-specific design choice. We'll build all three in this lesson:

Chapter 2: The Battle Loop — Anatomy of One Vote

Before any math, let's trace exactly what happens, end to end, when one user runs one battle. This is the data lifecycle of Figure 1 in the paper. The platform has three modular pieces that talk over plain HTTP: a Frontend (the web UI), a Backend (the orchestrator), and a set of Model Endpoints (the TTM systems, each in its own Docker container).

Each box is a stage in one battle. Click to inspect the data flowing through it — the prompt text, the structured route, the two audio blobs, and the final vote record.

Walking the flow with real tensor-level data

Let's trace the actual battle from the paper's appendix — prompt "Celtic punk song with prominent vocals and lyrics about an evaluation platform called Music Arena":

1. Prompt in. A single string: prompt = "Celtic punk song…". One text box. The user does not specify lyrics, duration, or whether it's instrumental.
2. Route. The backend calls an LLM (GPT-4o) which returns structured fields: {instrumental: false, lyrics: null, duration: null}. "Vocals" is implied by "song with prominent vocals," so only vocal-capable models are eligible.
3. Generate in parallel. Two eligible models are picked uniformly at random — here riffusion-fuzz-1-0 and acestep. The backend dispatches both at the same time, and crucially waits for both before showing anything.
4. Deliver simultaneously. The two audio blobs come back with metadata: A is 44.1 kHz, 2-channel, 192.5 s; B is 48 kHz, 2-channel, 30.0 s. The UI hides both the durations and the model names.
5. Listen + vote. The user plays A and B (timestamps logged), then votes preference: "A" with free-text feedback. Only now are the model identities and generation speeds revealed.
6. Store. The full record — salted user ID, prompt, both audios, listening timestamps, vote, feedback — is written to the database, ready for the leaderboard and the monthly data release.

Chapter 3: Bradley-Terry — From Clicks to a Number

We have a pile of pairwise votes: "A beat B," "C beat A," "B beat C"… How do we turn that into a single score per model that we can rank? We need a model of how a "win" is generated. The 1952 Bradley-Terry model is that model, and it's beautifully simple.

Manufacturing the need

Naive idea: just use raw win rate. Model A won 70% of its battles → score 0.70. Why is that broken? Because it ignores who you played. Beating a strong model 60% of the time is far more impressive than beating a weak model 60% of the time. Win rate has no notion of opponent strength. We need a score where each model has a hidden "strength," and the probability of winning depends on the difference in strengths.

Every symbol, with a plain analogy:

• β_i — model i's strength. Think of it as height on a "skill ladder." Only differences matter (you can add 100 to everyone and nothing changes).
• β_i − β_j — the gap. If A is one rung above B, A is favored; two rungs, heavily favored.
• σ — the logistic squash. It turns any real-number gap into a probability in (0,1). Gap of 0 → 0.5 (coin flip). Big positive gap → near 1. Big negative → near 0.

A worked numerical example

Suppose β_A = 1.0 and β_B = 0.0. Then the gap is 1.0, and:

So a model one strength-unit above another should win ~73% of head-to-head battles. Now invert it: if we observe A winning 73% of the time, the maximum-likelihood strength gap is ln(0.731/0.269) = ln(2.72) = 1.0. That inversion — going from observed win rates back to strengths — is exactly what fitting Bradley-Terry does. We choose the β values that make the observed battles most probable.

From strength to "Arena Score"

The raw β are tiny numbers around 0. To make them readable (and to match chess intuition), they're rescaled into an Arena Score on an Elo-like scale, anchored around ~1000–1200:

The factor 400/ln(10) ≈ 173.7 is the classic Elo convention: a 400-point gap means the stronger model wins ~10× as often (a 90.9% win probability). In the paper's first leaderboard, riffusion-fuzz-1-1 scored 1250.8 on instrumental music while sao-small scored 782.4 — a 468-point gap, i.e. the top model wins the head-to-head the overwhelming majority of the time.

Chapter 4: The Leaderboard Simulator — Watch Scores Converge

This is the heart of the whole platform, made interactive. Below are five fictional TTM models with hidden true strengths (which you set with the sliders). Press Run battles to draw random pairs, simulate votes using Bradley-Terry, and fit the Arena Score live. Watch the estimated scores climb toward the truth — and watch the 95% confidence intervals (the thin error bars) shrink as votes pile up.

The ghost outline on each bar is the true Arena Score implied by the slider. The filled bar is the score fitted from your simulated votes; the whisker is its 95% confidence interval. Drag two sliders close together, then run only 50 battles — see how the CIs overlap so you can't rank them. Run 1000 and they separate.

Try to break it

• Set two models equal. Their fitted scores will jitter around each other forever — no amount of data separates a true tie. That's correct behavior, not a bug.
• Give one model a huge lead. Its CI tightens fastest, because lopsided battles are more "informative" (a 99% win rate pins the gap quickly).
• Reset and run just 50. Notice the ranking can be wrong early. The paper's lowest-vote vocal model (preview-jerboa, 88 votes) has a CI of roughly ±85 points for exactly this reason.

Chapter 5: LLM Routing — One Text Box, Many Type Signatures

Here's a problem unique to music. In Chatbot Arena, every model has the same interface: text in, text out. You can route any prompt to any model. But TTM models are a zoo:

So if a user types "folk song about a cat named Chamomile," you can't send that to MusicGen (instrumental — no vocals). And you can't send it raw to SongGen either, because SongGen demands a lyrics field the user never provided. The single text box has to be adapted to each model's type signature.

Each prompt is parsed into structured fields, then matched against the model roster. Green = compatible (will be in the battle pool). Gray = filtered out. The copyrighted prompt is rejected at moderation before any model sees it.

The data-flow, precisely

Trace the types. Input: prompt: str. The LLM emits {instrumental: bool, lyrics: str|null, duration: float|null} plus a moderation verdict {allowed: bool, reason: str}. The router then computes, for each model, compatible = model.supports(parsed), and draws the battle pair uniformly from the compatible set. For a vocal prompt routed to a lyrics-required model like ACE-Step, the LLM is called a second time to synthesize lyrics — the missing field is filled in, not left null.

Chapter 6: Listening Data — Did They Actually Listen?

An image vote is honest by construction — you see the whole image at a glance before voting. Music isn't like that. It unfolds over time. A user could vote after hearing two seconds, or skip clip B entirely. A vote from someone who never heard B is garbage. Music Arena's response: measure the listening, and gate the vote.

The log is a list of (event, timestamp) pairs per clip. From the paper's example battle, clip A's log looks like:

Click Play A and Play B to accumulate listen time. The vote button unlocks only once both bars cross the 4-second threshold. This is the actual mechanism enforcing meaningful engagement.

Beyond the gate, users are asked for optional natural-language feedback — per-clip notes like the paper's real example: "The music quality was much higher overall, though the style wasn't quite what I asked for." Binary A-vs-B tells you which; the text tells you why, which is what you need to actually align a model.

Chapter 7: Privacy — Linking Users Without Knowing Them

Music Arena wants two things that sound contradictory. (1) Record linkage: recognize that 30 battles came from the same person — essential for spam detection and longitudinal "do preferences drift?" research. (2) Privacy: never store who that person actually is. How do you have a stable identity without storing the identity?

Symbol by symbol: IP is the raw identifier you must never store. salt is a secret random string known only to the server. ‖ is concatenation. HASH is a one-way function — easy forward, infeasible backward.

Why the salt is the whole game

Without a salt, an attacker mounts a rainbow table attack: there are only ~4 billion IPv4 addresses, so you precompute HASH(every_ip) once, then reverse any stored hash by lookup. Trivial. The secret salt defeats this — the attacker would have to recompute the entire table per salt, and they don't know the salt. The hash you publish is useless to anyone but the server.

Three users hit the platform repeatedly. Same IP → same anonymized ID (linkable). Change the salt and all IDs change at once — so even if last month's table leaked, this month's hashes are unrelated. The raw IP is never stored; only the right-hand column survives.

Chapter 8: The Speed–Quality Tradeoff (RTF)

A leaderboard with one number ("Arena Score") hides a decision every TTM provider must make. A model targeting consumption (you want a finished track) should maximize quality. A model targeting a creative workflow (a producer iterating live) should minimize latency, even at some quality cost. Ranking by score alone punishes the fast-and-good-enough model unfairly. So Music Arena reports a second axis.

Worked example: sao-small in the paper has RTF 12.79× (blazing fast, score 782) while riffusion-fuzz-1-1 has RTF 6.01× (slower, but score 1250). The fast model is roughly twice as fast yet ranks last on quality — a textbook speed-quality tradeoff. Neither number alone tells you which to pick; your application decides.

Each point is a real model from the paper's first leaderboard. X = median RTF (log scale, faster → right). Y = Arena Score. The frontier is the upper-right: high quality and fast. Toggle the highlight to see how open-weights vs proprietary models populate the space — and note no single model dominates both axes.

Chapter 9: Connections & Cheat Sheet

Music Arena is not a model — it's an evaluation methodology carefully adapted to a domain. Its lineage runs straight through the "Arena" family, and its real payload is the renewable preference dataset it produces.

Where it sits

• Chatbot Arena (2024) — the original: pairwise LLM battles → Bradley-Terry. Music Arena inherits the loop, adds music-specific machinery (routing, listening data, RTF).
• TTS Arena, GenAI Arena, 3D Arena, Copilot Arena — the same template ported to speech, images/video, 3D, and code. Music Arena's thesis: each domain needs tailored adaptation, not a copy-paste.
• The models being judged — MusicGen, Stable Audio Open, ACE-Step, SongGen, Riffusion FUZZ, Lyria/Magenta RealTime. Many are diffusion or autoregressive audio models worth their own lessons.
• The downstream use — preference data feeds RLHF-style alignment (cf. MusicRL) and trains better automatic metrics that finally correlate with humans.

Limitations the paper is honest about

• Pairs chosen uniformly at random — not adaptively (no active sampling of informative pairs), so low-vote models keep huge CIs.
• Can't track seeking within a clip — only total listen time, not which 5-second segment you replayed.
• Skewed population — mostly US users and AI enthusiasts, so the "human preference" is a particular slice of humanity.
• Sustainability — free self-hosted open-weights inference costs money; who pays for it long-term is unsolved.

Cheat sheet — every key equation