StylePitcher: Style-Following
Pitch Curves for Singing

Chapter 1: The Key Insight

StylePitcher's central trick is to refuse to build a "pitch predictor." Instead it reframes pitch generation as masked infilling — exactly the way a language model fills a blank in "the cat sat on the ___", but over a continuous pitch signal.

Given a full pitch curve, you hide a stretch of it with a binary mask. The visible part is the context; the hidden part is what the model must regenerate. Because the context comes from the same singer, the model learns — implicitly, with no singer labels — to continue that singer's vibrato, slides, and intonation into the masked region while still landing on the target notes.

The two-part recipe

Each design choice removes one specific failure from Chapter 0:

What the whole model is, in one breath

StylePitcher is: a diffusion transformer (DiT) trained with rectified flow matching to denoise a pitch curve, conditioned on the target notes, the visible context (unmasked F0), and an unvoiced indicator (where there's no pitch, e.g. silence or consonants). It learns to fill masked frames so they follow the context's style and hit the score. At inference you integrate a learned velocity field for K=16 steps to turn noise into a finished curve. We'll build every piece — flow matching, the mask, the DiT, the smoothing of the score, and the sampler — from scratch.

Chapter 2: Pitch Curves — What We're Modeling

Before any architecture, be precise about the signal. A pitch curve (F0 curve) is, at each 20 ms frame, the fundamental frequency the voice is producing — or a flag saying "no pitch here" (a breath, a silence, an unvoiced consonant like "s"). It is the difference between a robot reading notes and a human singing them.

The expressive techniques hiding in the wiggle

A flat line at the note frequency would sound dead. Real singing rides on top of the target note with structured deviations:

• Vibrato — a periodic oscillation around the note, typically 5–7 Hz, that adds warmth. Its rate and depth are personal.
• Pitch slides / glissando — the voice sweeps from one note into the next rather than jumping instantly.
• Overshoot & preparation — the voice often overshoots a target note then settles, or dips before rising into it.
• Microtonal intonation — landing a few cents sharp or flat, a stylistic signature, not an error.

The dashed line is the symbolic score — the flat target notes (in semitones). The orange curve is an expressive F0 curve riding on top. Crank vibrato and slide to hear (with your eyes) the difference between "the notes" and "a performance." StylePitcher's whole job is generating that orange curve from the dashed one — in the style of a given singer.

The data the model actually sees

Three aligned sequences, all length N (up to 1024), define one training example:

From these the model defines context x_ctx = (1−m) ⊙ x (the visible part) and target x_mask = m ⊙ x (what to predict). The ⊙ is elementwise multiply — masking is just zeroing out the frames you want to hide.

Chapter 3: Rectified Flow — The Engine

We need a generative model for the continuous F0 signal. StylePitcher picks rectified flow, a streamlined relative of diffusion. The promise: turn random noise into a realistic pitch curve in a handful of steps, with a remarkably simple training objective. Let's build the intuition before the formula.

Manufacture the need first

Diffusion models learn to denoise: start from pure noise and gradually remove it. They work, but the standard formulation involves a curved, stochastic path from noise to data and often needs dozens to hundreds of steps. The question rectified flow answers: what is the simplest possible path from noise to data, and can we learn to travel it?

The answer is almost embarrassingly direct: draw a straight line between a noise sample and a data sample, and teach a network to follow that line.

The three equations, derived

Rectified flow learns a time-dependent velocity field v_θ(x_t, t, c) — a neural net (weights θ) that says "if you're at point x_t at time t under conditions c, here's the direction and speed to move." Generation is solving an ODE (ordinary differential equation):

Symbol by symbol: x_t is your current point (a pitch curve being formed); t ∈ [0,1] is a scalar clock from noise (0) to data (1); c bundles the conditions (notes y, context x_ctx, unvoiced u); v_θ is the velocity to integrate. The interpolation that defines the training path:

where x₀ ∼ π₀ is Gaussian noise N(0,1) (called ε in the paper) and x₁ ∼ π₁ is a real pitch curve. Differentiate that line w.r.t. t: dx_t/dt = x₁ − x₀. That constant is the target velocity. So we train the network to match it:

Read it as: sample a noise point, a real curve, and a random time; build the interpolated point x_t; ask the network for its velocity there; penalize the squared distance to the true straight-line velocity x₁−x₀. That's the entire training loss. No noise-schedule bookkeeping, no variational bounds.

A worked numerical step

Take a single scalar to see it concretely. Noise x₀ = −1.2, data x₁ = 0.8, time t = 0.25:

So at the point x_t=−0.7 (clock 0.25), the network should output velocity 2.0. If you took an Euler step of size Δt = 0.75 from here: −0.7 + 2.0×0.75 = 0.8 = x₁ — you land exactly on the data. The straight path makes the math fall out cleanly.

Left dots = Gaussian noise x₀; right dots = a real pitch curve x₁. The green points show the interpolation x_t as you drag t from 0 to 1 — every point travels a straight line at constant velocity x₁−x₀ (the gray arrows). That constant velocity is exactly what the network is trained to predict.

Chapter 4: The Infilling Formulation

Now we fuse the flow engine with the masked-infilling idea. The model defines a conditional distribution over the masked frames:

Read it: predict the hidden pitch x_mask = m⊙x, given the complete note sequence y and the visible context x_ctx = (1−m)⊙x. This is borrowed from Voicebox (the speech equivalent). The magic is "through in-context learning, the generated segments implicitly follow the singing style of the surrounding context while remaining aligned with the target score."

The training objective, with the mask folded in

Take the rectified-flow loss from Chapter 3 and (1) add the conditions, (2) gate it by the mask:

Symbol check: ε ∼ N(0,1) is the noise (this paper's x₀); x is the real curve (the x₁); so (x−ε) is the same straight-line velocity as before. The m⊙ in front means: only the masked frames contribute to the loss. Everywhere else the gradient is zero — those frames are given as context, there's nothing to learn there.

Teal = visible context (a singer's real curve, full of vibrato). The shaded gap is the mask. Hit Generate fill and watch the orange curve fill the gap — continuing the context's vibrato while tracking the dashed target notes. Drag the mask to the very end and it becomes a continuation task (pitch correction); put it in the middle and it's an inpainting task. Same model, just a different blank.

The core insight as runnable code

# Rectified-flow masked-infilling loss for pitch curves
import torch

def flow_infill_loss(net, x, y, u, mask):
    """x:[B,N] log-F0   y:[B,N] notes   u:[B,N] unvoiced   mask:[B,N] (1=predict)"""
    eps = torch.randn_like(x)                 # x0 ~ N(0,1)
    t   = torch.rand(x.shape[0], 1)            # clock in [0,1], per example
    x_t = (1 - t) * eps + t * x            # straight-line interpolation
    x_ctx = (1 - mask) * x                # visible context (masked frames zeroed)
    v = net(x_t, t, y, x_ctx, u)              # predicted velocity, full sequence
    target = x - eps                          # constant straight-line velocity
    # loss only on masked frames
    return ((mask * (v - target)) ** 2).sum() / mask.sum().clamp(min=1)

Chapter 5: The Diffusion Transformer

The velocity field v_θ is a diffusion transformer (DiT) — 8 layers, 8 heads, 512 hidden dim, rotary position embeddings, 49M parameters total. Now let's trace exactly what tensors flow in and how they're fused, because "what shape goes where" is the whole engineering story.

The data flow, shape by shape

Three streams enter, each length N (the frame axis):

These three embeddings are concatenated along the feature dimension — [N, 512+512+256] = [N, 1280] — then linearly projected down to the transformer's working width for the next layers. Additionally:

• The unvoiced indicator u ∈ {0,1}^N is incorporated so generated F0 aligns with sung phonemes.
• The flow step t is sinusoidally encoded (like a positional encoding for the scalar clock) and used to modulate the representation — this is the "DiT" signature: the timestep doesn't just get concatenated, it conditions the layer normalizations / scales, telling every layer "we're at noise level t."
• Rotary position embeddings (RoPE) give the transformer a sense of which frame is which along the time axis.

Why 512 for pitch/context but only 256 for notes?

The notes y are coarse: one of 72 semitone classes per frame — low information. The continuous F0 values (x_t, x_ctx) carry the fine expressive detail the model must reproduce, so they get double the embedding width. It's a budget allocation: spend representation capacity where the signal is rich.

Chapter 6: Training — CFG, Masking, Schedule

Training a flow model that's controllable takes three ingredients beyond the basic loss. Each fixes a specific problem.

1. Classifier-free guidance (CFG): learning to ignore conditions on purpose

To later amplify the influence of the notes and context at inference, the model must also know what generation looks like without them. So during training, the conditions y, x_ctx, u are randomly dropped (replaced by a null token ∅) with probability p_c = 0.5. Half the time the model practices unconditional generation; half the time conditional. This gives us two velocity fields to combine at sampling time (Chapter 8).

2. Masking schedule: mostly-blank, so it learns to generate, not copy

The fraction of frames masked is drawn r ∼ Uniform[70%, 100%] per example. That is aggressive — usually 70–100% of the curve is hidden. Why so high? If you only masked 10%, the model could cheat by interpolating between nearby visible frames and never learn to generate long expressive stretches. Forcing it to fill huge gaps from sparse context teaches genuine style continuation.

3. Cosine time schedule: spend effort where denoising is hard

Instead of sampling the flow time t uniformly, a cosine schedule biases toward lower t (closer to noise). Early in the trajectory (near pure noise) the model has the least information and the hardest job — so it gets more training samples there. It's the same logic as diffusion noise schedules: allocate training where the signal-to-noise ratio is worst.

The full training recipe (Concept + Realization)

Toggle between the two sampling distributions used in training. Mask ratio is uniform over 70–100% (the curve is mostly hidden). Cosine time schedule piles probability near t=0 (pure noise) where denoising is hardest. Both are deliberate biases that focus learning on the hard cases.

Chapter 7: The Smoothing Algorithm

One quietly crucial contribution. The model needs a clean symbolic score y as a condition — "which note should be sung here." But where does that score come from? You can't rely on hand annotations for 1916 hours of karaoke. So the authors extract it automatically with a MIDI transcriber (Basic Pitch). And here's the trap.

The fix: blur the score, keep the notes

The transcriber produces a multi-pitch activation map — a 2-D heat map of (time × pitch) showing how strongly each pitch is active at each moment. Vibrato shows up as a fuzzy band oscillating up and down; the underlying note is the central tendency. So:

1. Apply a Gaussian blur across the activation map. This smears out the fast vibrato oscillation (high-frequency wiggle) while preserving the slow, sustained note (low-frequency structure). It's a low-pass filter on the score.
2. Post-process: remove very short rests and very short notes (transcription noise).

The result is a clean, blurred score that says "this phrase is a C, then a D" — without saying "...with this exact vibrato," because generating the vibrato is the model's job, conditioned on the singer's style context.

The orange wiggly line is the raw transcribed pitch (vibrato and all). The teal line is after Gaussian blur with width σ. Crank σ and watch the vibrato dissolve while the underlying note steps survive. With Quantize on, you see the final clean score (the dashed steps) — that's y. Set σ=0 to see the polluted score the model must not be conditioned on.

Chapter 8: Sampling — Integrating the ODE (the showcase)

Training taught the network a velocity field. Generation is the payoff: start from pure noise and follow the arrows from noise to a finished pitch curve, while steering with classifier-free guidance.

The guided velocity

At inference we don't just use the conditional velocity. We combine it with the unconditional one to amplify the conditions, using CFG scale α:

Decode it. v_θ(·,∅) is the unconditional velocity (conditions dropped). v_θ(·,y,x_ctx) is the conditional one. Their difference is "the direction the conditions push you." Multiplying that push by α = 1.25 (the paper's value) and adding it back means: go where you'd go anyway, then take 1.25× the conditional nudge. α=1 is plain conditional; α>1 sharpens adherence to the notes and context.

Integrating: K=16 midpoint steps

To turn noise into data we numerically solve dx_t = v̂_θ dt from t=0 to t=1. The paper uses the midpoint solver with K=16 steps. Midpoint (a 2nd-order Runge–Kutta method) evaluates the velocity at the step's midpoint for a more accurate move than plain Euler — buying high quality with very few steps, which is the whole appeal of rectified flow vs. many-step diffusion.

Hit Run sampler to integrate the ODE from noise (jagged gray) toward a finished pitch curve (orange), tracking the target notes (dashed). Drag steps K down to 2–3 and watch quality degrade (the straight-path assumption breaks with too few steps). Drag CFG α up to over-sharpen toward the notes (style flattens); down toward 0 and the curve drifts off the score. The paper's sweet spot is K=16, α=1.25.

# CFG-guided rectified-flow sampling (midpoint solver)
import torch

@torch.no_grad()
def sample_pitch(net, y, x_ctx, u, N, K=16, alpha=1.25):
    x = torch.randn(1, N)                       # start at noise (t=0)
    dt = 1.0 / K
    def vhat(x, t):
        v_uncond = net(x, t, None, None, None)    # conditions dropped
        v_cond   = net(x, t, y, x_ctx, u)
        return v_uncond + alpha * (v_cond - v_uncond)
    for i in range(K):
        t = torch.full((1,1), i * dt)
        k1 = vhat(x, t)                         # slope at start
        k2 = vhat(x + 0.5*dt*k1, t + 0.5*dt)  # slope at midpoint
        x = x + dt * k2                     # midpoint step
    return x                                 # finished pitch curve (t=1)

Chapter 9: Three Applications, One Model

Here's the payoff of the infilling framing. All three singing tasks are the same model with a different concatenation-and-mask pattern. No retraining. Let's trace each one's data flow precisely.

1. Automatic Pitch Correction (APC)

You have off-key singing (F0 x_off, notes y_off, unvoiced u_off) and the target notes y_in it should have hit. Build the input as x = (x_off, 0) — the real off-key curve followed by a blank — with notes y = (y_off, y_in) and u = (u_off, u_off). Generate; take the latter (blank) half of the output as the corrected curve x̂_in. The off-key half acts as style context; the model produces an in-tune version in that singer's style — it corrects the notes while keeping the singer's slides and vibrato.

2. Zero-Shot SVS with Style Transfer

You have a reference performance (x_ref, y_ref, u_ref) and target content from an SVS model (x_tgt, y_tgt, u_tgt). Concatenate all three sequences, mask the target segment x_tgt, and regenerate it. The output x̂_tgt follows the reference's style while matching the target's notes — then replaces x_tgt in the synthesizer. You get a sung phrase with the right melody but a chosen singer's expressive flavor.

3. Style-informed SVC

Standard voice conversion keeps the original (or key-shifted) F0 — converting only timbre, losing the source singer's expressive pitch style. StylePitcher instead concatenates reference and target x = (x_ref, x_tgt), masks and regenerates x_tgt, so the converted singing transfers both timbre and pitch style while preserving content.

Each task is the same input layout: a sequence of frames where teal = given context (supplies style), gray = masked region the model fills, and the dashed line = the target notes for the filled region. Switch tasks and watch only the mask placement and which notes go where change. That is the entire reconfiguration — no weights move.

Chapter 10: Experiments & Results

The claim to validate: StylePitcher matches or beats task-specific baselines on pitch accuracy while clearly winning on style capture — without ever being trained for those specific tasks.

Objective metrics (GTSinger, unseen by all models)

RPA/RCA/OA measure pitch alignment (how well the curve hits the right notes; higher is better). Acc. is a sneaky style metric: train an LSTM to tell generated curves apart from real ones — lower accuracy means the generated curves are more indistinguishable from real, i.e. more natural/expressive.

The headline number: Acc. = 51.85% — near random (50%). The LSTM essentially cannot tell StylePitcher's curves from real human ones, versus ~69–71% for baselines. That's the rectified-flow continuous modeling paying off: the curves look real.

Subjective metrics (human listening test, 3 tasks)

MOS-P (pitch), MOS-S (style), MOS-Q (quality), 5-point scales, 19 listeners.

Switch tasks to compare StylePitcher (orange) against the task-specific baseline (teal) on Pitch, Style, and Quality. The pattern: StylePitcher trades a little pitch-correction strictness (it deliberately keeps expressive slides instead of snapping to the grid) for clearly better style capture across all three tasks.

What the figures show qualitatively

• APC: StylePitcher keeps expressive pitch slides while correcting notes — "more personalized corrections rather than enforcing strict alignment." It fixes the tuning without sterilizing the performance.
• SVS: It captures vibrato and glissando that StyleSinger misses — despite never being trained jointly with the synthesizer.
• SVC: It transforms a flat target curve into one with strong vibrato lifted from the reference — transferring style, not just timbre.

Honest limitations the paper admits

• Content-unaware style can sound unnatural. In SVC, "applying expressive techniques without content awareness can occasionally produce unnatural results" — e.g. heavy vibrato landing on a syllable where it doesn't belong. They flag this for future work.
• Lower pitch-correction strictness. By design it preserves slides, so MOS-P for correction is below the strict baseline. Whether that's a bug or a feature depends on taste.
• Small subjective study. 19 listeners, 76 ratings per condition — directional, not definitive.

Chapter 11: Connections & Cheat Sheet

StylePitcher sits at the confluence of three lineages: the masked-infilling-as-generation idea from speech (Voicebox), the rectified flow generative engine, and the long tradition of singing-AI systems that all need a pitch curve.

The cheat sheet — every key equation

Why StylePitcher matters

• It unifies singing tasks. One frozen model + a mask layout = correction, synthesis, or conversion. The task is the mask.
• It models style without labels. No singer embeddings or IDs — style is read from context, so it generalizes to unseen voices.
• It separates control from generation. The smoothing algorithm is a reusable principle: condition on what to control, never on what to generate.
• It demonstrates rectified flow on 1-D signals. Near-indistinguishable-from-real curves in ~16 steps — a template for any continuous-signal generation.

The one-paragraph summary you could give on a whiteboard

StylePitcher generates expressive, singer-specific pitch curves by framing the problem as masked infilling: hide part of an F0 curve, and a rectified-flow diffusion transformer regenerates it conditioned on the target notes, the visible context (which implicitly supplies the singer's style), and an unvoiced indicator. Rectified flow learns straight noise→data paths via a velocity-matching loss, so a midpoint solver produces realistic curves in ~16 steps; classifier-free guidance (α=1.25, trained by 50% condition dropout) steers adherence to the score. A Gaussian-blur smoothing of auto-transcribed MIDI cleanly separates the note (control) from the vibrato (to be generated). Because the task is just where you put the mask, the same frozen 49M-param model serves pitch correction, style-transfer synthesis, and style-informed voice conversion — matching task-specific baselines on pitch accuracy while producing curves an LSTM can barely distinguish from real ones (Acc. 51.85%).