Music-Aware Virtual Assistants
Notifications that Sing

Chapter 1: The Key Insight

The idea in one sentence: turn the notification text into a sung voice whose melody and timing agree with the song, then temporarily swap that sung voice in for the song's original vocals.

Read it again, because two distinct moves are hiding in there. First, generation: you must produce a singing voice that says the notification words but obeys the song's musical rules. Second, substitution: you must place that voice into the music where a voice already lives — the vocal track — so the listener hears one coherent performance, not a duet of strangers.

Why this beats the alternatives the authors considered

Imagine the design space. You could (a) play a spoken notification over the music — but then two unrelated audio streams fight, and the speech is hard to parse. You could (b) duck the music and speak — the conventional baseline, jarring. Or you could (c) generate a sung notification and layer it on top — but now you have two voices, the original singer and the notification, talking over each other.

Option (d), the paper's choice, is to generate a sung notification and remove the original vocal during it. One voice at a time, musically consistent. That single decision — replace, don't overlay — is what makes the result feel like part of the song.

The pipeline in one breath

Everything downstream is the realization of that insight. We will build each box from scratch:

Chapter 2: Music 101 — Just Enough Theory

Before we can analyze a song or compose a melody, we need a precise vocabulary for three things the system manipulates: pitch, scale/key, and beat. If you already know music, skim; if you don't, this is all you need.

Pitch is log-frequency

A musical note is a frequency in Hz, but our ears hear pitch logarithmically: doubling the frequency raises the pitch by one octave, and that feels like an equal step whether you start low or high. So music uses a log scale. The standard mapping from a MIDI note number n to frequency is:

Symbol by symbol: n is an integer note number (60 = middle C, 69 = A4), 440 Hz is the reference pitch A4, and 12 is the number of equal half-steps (semitones) per octave. The 2^1/12 factor is one semitone — the smallest step on a piano. Move up 12 of them and frequency doubles.

A key is a 7-note menu carved from the 12

There are 12 pitch classes per octave (C, C#, D, ... B). A key (say "C major") picks 7 of them as "in-bounds" — the notes that sound consonant together. The C-major scale is the white keys: C D E F G A B. Sing any of those over a C-major song and it fits; sing a black key and most people wince. This is the single most important constraint on our melody.

Beat is the clock you tap your foot to

Underneath a song is a steady pulse — the beat. Its rate is the tempo, measured in beats per minute (BPM). At 120 BPM there is a beat every 0.5 s. Lyrics in real songs almost always start syllables on or near beats; that alignment is what makes singing feel "in the pocket." Our notification must do the same: each syllable starts on a beat.

A piano octave. Highlighted keys are in the chosen scale (singable); dim keys are out. Change the root and watch the 7-note menu slide. The frequency under each highlighted key is computed from the formula above.

Chapter 3: Music Analysis — Reading the Song

The system receives only the audio of whatever the user is playing — no score, no metadata. So the first job is to listen: recover the beat grid and the key from the raw waveform. These are two of the oldest tasks in MIR, and the system uses well-established estimators (the paper builds on standard music-analysis tooling rather than inventing new detectors).

Beat tracking, from intuition to algorithm

Where are the beats? The classic recipe has three steps. (1) Compute an onset strength signal — at each instant, how much new energy just appeared (a drum hit, a strummed chord). You get this from the frame-to-frame increase in spectral energy. (2) Find the dominant period of that onset signal — its autocorrelation peaks at the beat period, giving tempo. (3) Lay down a grid at that period and slide it to line up with the strongest onsets.

Here |X(t,f)| is the magnitude of the spectrogram at time frame t and frequency bin f. The max(0, ·) keeps only increases in energy (note onsets), discarding decays. Sum over all frequencies and you get one number per frame: "how much started right now." Peaks in this curve are candidate beats.

Key detection via pitch-class profiles

To find the key, count how much energy each of the 12 pitch classes receives across the song — a 12-bin histogram called a chroma or pitch-class profile. A song in C major spends most of its energy on C, E, G and the other white keys. Then correlate that 12-bin profile against a stored template for each of the 24 possible keys (12 roots × major/minor); the best-matching template wins.

The orange curve is a synthetic onset-strength signal — spikes are note onsets. Drag Tempo to change the underlying pulse; drag Noise to muddy the signal (weak onsets, distractor peaks). Toggle the beat grid (teal lines) to see where the tracker would place beats. Watch how high noise makes the grid drift off the true onsets — the failure mode of beat tracking.

# Beat + key from raw audio with librosa (the standard MIR toolkit)
import librosa, numpy as np

y, sr = librosa.load("user_song.wav")            # waveform [T], sample rate

# --- beats ---
onset = librosa.onset.onset_strength(y=y, sr=sr)    # [frames]: energy increases
tempo, beat_frames = librosa.beat.beat_track(onset_envelope=onset, sr=sr)
beat_times = librosa.frames_to_time(beat_frames, sr=sr)  # [N] timestamps (s)

# --- key (via chroma / pitch-class profile) ---
chroma = librosa.feature.chroma_cqt(y=y, sr=sr)     # [12, frames]
profile = chroma.mean(axis=1)                     # [12] avg energy per pitch class
# correlate `profile` against 24 Krumhansl key templates -> argmax = key
key = best_key(profile)                             # e.g. "A minor"
print(tempo, beat_times[:4], key)               # 120.0 [0.46 0.95 1.44 ...] A minor

Chapter 4: Melody Fitting (the showcase)

Now the creative heart. We have the song's key (a 7-note menu) and its beat grid (timestamps). We have notification text — say "Meeting in ten minutes". We must produce a melody: an assignment of each syllable to a pitch (from the key) and a start time (on a beat) and a duration. This melody must agree with the song and be singable for the words.

Step 1 — text becomes syllables

You sing syllables, not letters. So first split the text into syllables: "Mee-ting in ten mi-nutes" → 6 syllables. Each syllable will get one note (held notes can span more, but one-note-per-syllable is the clean baseline). The count of syllables tells you how many notes — and beats — you need.

Step 2 — syllables snap to beats

Lay the 6 syllables onto consecutive beats from the grid. If the song is 120 BPM, beats are 0.5 s apart, so the notification spans 6 × 0.5 = 3 seconds. The tempo of the song sets the pace of the notification automatically — no separate "speech rate" knob. This is the elegant part: by snapping to beats, pace and rhythm both come for free from Ch 3's grid.

Step 3 — choose in-key pitches

Each note's pitch must come from the song's key (the 7 in-bounds notes). A natural, melodic choice is to take small steps — mostly stay on a note or move to a neighbor in the scale, occasionally leap — because human melodies are locally smooth. A purely random in-key walk already sounds plausibly musical; biasing toward small intervals sounds intentional. The constraint "stay in key" does the heavy lifting; the smoothness bias adds polish.

A piano-roll. The x-axis is time (vertical lines are beats); the y-axis is pitch. Each block is one syllable of "Mee-ting in ten mi-nutes". Rows that are part of the key are tinted. Turn In-key OFF to allow out-of-scale notes (watch blocks land on red rows — that is the "sour" failure). Drag Smoothness toward 0 to allow wild leaps (random/atonal); toward 100 for a smooth, intentional line. Recompose rolls a new melody under the current constraints.

The core insight as runnable code

import random

MAJOR = [0,2,4,5,7,9,11]        # semitone offsets of a major scale

def compose(text, root_midi, beat_times, smoothness=0.8):
    """Map each syllable to (pitch, start_time) obeying key + beat + smoothness."""
    syllables = syllabify(text)                 # ["Mee","ting","in","ten","mi","nutes"]
    scale = [root_midi + s for s in MAJOR + [12+x for x in MAJOR]]
    notes, idx = [], scale.index(root_midi)        # start on the tonic
    for i, syl in enumerate(syllables):
        notes.append((scale[idx], beat_times[i], syl))   # pitch ON a beat
        # next pitch: small step (smooth) vs occasional leap
        step = random.choice([-1,0,1]) if random.random()else random.choice([-3,-2,2,3])
        idx = max(0, min(len(scale)-1, idx+step))   # stay inside the scale
    return notes   # [(pitch, start_s, syllable), ...] -> feed to the singer

Trace the data: in is (text, key, beat_times); out is a list of (pitch, start_time, syllable) triples — exactly the score the singing synthesizer (Ch 5) consumes. Notice idx is clamped to the scale array, so we cannot leave the key; and beat_times[i] guarantees the i-th syllable starts on the i-th beat. The two hard constraints are structurally enforced; only the pitch contour is random.

Chapter 5: Singing Synthesis — From Score to Voice

We have a score: (pitch, start, syllable) triples. Now we need an actual singing voice waveform that says those words at those pitches and times. This is singing voice synthesis (SVS) — text-to-speech's musical cousin, with an extra input the TTS world ignores: a target pitch (F0) contour the voice must follow.

Why you cannot reuse plain TTS

A normal TTS engine speaks. Its prosody — the natural rise and fall of pitch — is whatever the model learned for speech, and you cannot dictate "hold this exact note for this exact duration." But our whole premise is pitch control: the voice must sit on the scale notes from Ch 4. So the synthesizer must accept three aligned inputs: the phonemes (what to say), the F0 contour (what pitch to say it at), and the durations (how long each note lasts, set by the beat grid).

The simplest realization: synthesize, then pitch-correct

A practical route — and the easiest to reason about — is two stages. (1) Run ordinary TTS on the notification text to get a spoken waveform. (2) Pitch-shift each syllable to the target note using a phase vocoder or PSOLA, snapping the speech's natural pitch to the melody. Stretch or compress each syllable's duration to fill its beat. The result is "spoken words, sung pitches" — robotic if done crudely, but the principle is clear and it isolates pitch control as the new ingredient over TTS.

More modern SVS models (neural, end-to-end) take the phoneme + F0 + duration inputs directly and render a far more natural singing voice in one pass — but the interface is the same. What matters for understanding the paper is the contract: score in, sung waveform out, with pitch dictated by us.

The gray line is the natural speech pitch (TTS prosody) for each syllable. The teal dashes are the target melody notes from Ch 4. Drag Shift strength: at 0 you hear plain speech (pitch ignores the melody); at 100 each syllable is fully snapped to its target note (singing). Step through to advance syllable by syllable and watch the correction.

Chapter 6: Vocal Swap & Mixing

We have a sung notification waveform that matches the song's key, beat, and pace. The final move from Chapter 1: replace the song's original vocal during the notification window and mix our voice into that slot — so the listener hears one coherent performance.

Step 1 — suppress the original vocal

To swap voices we must first remove the original singer for those few seconds. This is music source separation: split the mix into stems — typically vocals, drums, bass, other. Modern separators (e.g. Demucs / Spleeter-style models) take the stereo mix and output a vocals stem and an accompaniment stem. We keep the accompaniment (the instrumental bed) and discard the vocal stem only within the notification window; outside it, the original song plays untouched.

Step 2 — mix into the slot

Now add the sung notification to the (vocal-suppressed) accompaniment. The mix at each instant in the window is:

where accomp(t) is the instrumental bed, sung(t) is our notification voice, and g is a gain that places the voice at a believable lead-vocal level. To avoid clicks, the edges of the window are crossfaded: the original vocal fades out as the notification fades in, and back at the end.

Data flow, end to end of this stage

Stacked tracks over time: original vocal, accompaniment, and the notification. The shaded band is the notification window. Toggle Remove original vocal to see the vocal carved out inside the band (the paper's approach) vs left in (clashing duet). Hit baseline duck to see the conventional alternative — the entire song's level collapses and speech plays over silence. Drag gain to set the notification's level in the mix.

Chapter 7: The Full Pipeline

Time to assemble. Click each box to trace exactly what tensor or object flows between stages. This is the entire system the paper builds — every box is a chapter you have already seen.

The two inputs and the one output

The system has exactly two inputs — the user's music (a waveform) and the notification text (a string) — and produces one output: a blended mix where the music briefly sings the notification. Everything in between is the realization. Walking the data:

1. Music analysis (Ch 3): waveform → (key, beat_times).
2. Melody fitting (Ch 4): (text, key, beat_times) → score = list of (pitch, start, syllable).
3. Singing synthesis (Ch 5): (phonemes, F0, durations) from the score → sung waveform.
4. Source separation (Ch 6): mix → (vocals, accompaniment).
5. Mix + crossfade (Ch 6): accomp + g·sung in the window → blended output.

Chapter 8: The User Study

An HCI paper lives or dies by its evaluation, and here the "metric" is human preference. The authors ran a user study comparing their music-aware (sung) notification against the conventional approach — ducking the music and speaking the notification.

What they measured

Participants experienced notifications delivered both ways over real music and reported their reactions. The dimensions that mattered were intrusiveness (how much it disrupted the listening experience), blending (how well the notification fit with the music), and overall preference / delight.

Why "delight" is a legitimate result, not fluff

It is tempting to dismiss preference data as soft. But the entire premise of a notification UI is that humans tolerate it. A notification that users find less annoying is one they are more likely to leave enabled and actually heed — which is the real-world success criterion. The study also surfaces the tension every notification designer faces: noticeability vs. intrusiveness. A sung notification is gentler, but does it still reliably grab attention? That trade-off is the design space the next chapter maps.

Illustrative comparison of the two conditions across the study's dimensions (directional, reconstructing the paper's qualitative finding — the sung condition wins on blending and delight, and is less intrusive). Toggle the dimension. For Intrusiveness, lower is better.

Chapter 9: Failure Modes & the Design Space

A system this composite has many places to fail, and naming them sharpens your understanding of what each stage is really for. Honest engineering means knowing where the cracks are.

Where it breaks

• Wrong key → sour everything. If Ch 3 mis-estimates the key (common for ambiguous or modulating songs), every composed note is off-scale. The most catastrophic single failure, because key gates listenability.
• Beat drift → sloppy timing. Noisy or syncopated music defeats beat tracking; syllables land off the pulse and the notification feels unmusical. You saw this directly in the Ch 3 sim.
• Imperfect source separation → ghost vocals. Separators leak — bits of the original singer bleed through, so you faintly hear two voices despite the swap. Separation quality directly bounds how clean the swap sounds.
• Long notifications → awkward fit. A 25-word notification cannot gracefully fit a short musical phrase. The approach suits short, glanceable messages, not paragraphs.
• Intelligibility cost. Singing words is harder to parse than speaking them. If the user must catch every word (e.g. an address), gentle-but-mumbled may lose to clear-but-jarring.

The deeper point: notifications are a modality choice

Step back. The lasting idea here is not "singing assistants" specifically — it is that how information reaches you should be sensitive to what you are already doing. You are in a musical context; the notification should respect it. That reframing — context-aware delivery, not one-size-fits-all speech — is the seed the paper plants for future "ambient" assistants.

Chapter 10: Connections & Cheat Sheet

This paper sits at the crossroads of music information retrieval, audio synthesis, and HCI. Here is where its pieces connect to the rest of the field.

What each component descends from

The cheat sheet

Related lessons on Engineermaxxing

• WaveNet — the neural audio synthesis machinery behind modern singing/TTS voices.
• Diffusion & flow-matching audio models — the current frontier for generating singing voices end-to-end.
• Music source separation deep-dives — the Demucs-style stem splitters this system relies on.