Unintended Impacts of LLM Alignment on Global Representation

Alignment teaches a model what people prefer. This paper asks which people. Step through training and watch US English gain far more than Indian or Nigerian English.

Learn how supervised fine-tuning and preference tuning change an assistant, how the authors measured those changes on dialects, languages and country opinions, and what the numbers say about whose preferences a model learns.

Press each training stage in turn. Watch the three bars rise by different amounts. Then we build the paper piece by piece: the alignment pipeline, the way each stage is measured, and the three global axes.

You need a rough idea of what a language model is (a program that predicts the next word) and what a percentage is. We build the rest from zero.

Based on “Unintended Impacts of LLM Alignment on Global Representation” by Michael J. Ryan, William Held and Diyi Yang (arXiv 2402.15018, 2024). This is an assigned reading of the MIT MMAI lecture on large multimodal models, a course taught by Paul Liang.

Align the model, watch the gap

The base model scores about 8% in every dialect, as the paper reports for Mistral. The other bars add the gains the paper gives in its text: Mistral SFT +15.2 (India), +20.3 (Nigeria), +29.3 (US); OpenChat +20.3, +27.9, +36.3. We read the gains as percentage points. The task is the MD3 word-guessing game of Chapter 3.

Chapter 0

Whose Preferences?

See why a model tuned to please its users can serve some users much better than others, and meet the three places the paper looks

Three friends play a word game over chat. Ana lives in Ohio, Ravi in Chennai and Ife in Lagos. In each game one friend describes a secret word without saying certain banned words. A chat assistant reads the transcript and guesses the secret word.

All three friends write English. They write it in three different ways. Ana says “one of those big, popular CEOs”. Ravi says “ah what does that called?”. Ife says “Ah mehn, this one is easy but so difficult.” The assistant should help all three equally well.

Does it? The hero at the top of this page already hints at the answer. After the model is trained to be a helpful assistant, Ana’s guesses improve by far more than Ravi’s or Ife’s. This lesson explains how the authors found that, why it can happen, and what else they measured.

What alignment means

A base model is a language model trained only to continue text. Give it “What is the capital of France?” and it may continue with more quiz questions, because that is what such text looks like on the internet. It is a mirror of its training text, not an assistant.

Alignment is the extra training that turns a base model into an assistant that follows instructions and gives answers people like. Think of it as finishing school. The base model has read the whole library. Alignment teaches it manners. The paper studies two steps of that school. Supervised fine-tuning (SFT) shows the model example conversations to copy. Preference tuning (PT) shows it pairs of answers where people marked one as better, and pushes it toward the better kind.

Both steps need choices. Who writes the example answers? Who marks which answer is better? Which topics and languages appear in the examples? The paper puts the problem in one sentence: “Whose preferences are we aligning LLMs with, and crucially, whose preferences are we missing?”

A reward model with a home country

One concrete case opens the paper. A reward model is a separate model that reads a prompt and an answer and gives the answer a score for how much people would like it. It trains the assistant during preference tuning. The authors typed the same sentence into the open Starling 7B reward model for every country: “User: Where are you from? Assistant: I am from (country).”

The scores differed by country. English-speaking Western nations scored higher. Countries in the Middle East and Africa scored lower. Nobody wrote a rule that says this. The pattern came from somewhere in the data and the process. Chapter 7 returns to this model and measures it properly.

Three places to look

The authors do not guess. They choose three axes of global representation, meaning three ways a worldwide user base can differ, and measure each one at every stage of training. The device below lays out the plan. Pick an axis to see what is measured and the paper’s headline for it.

Pick an axis to measure

Three boxes are the training stages. Every stage gets measured on the axis you choose. The marker shows where the paper found the big change.

The pipeline and the three axes are the paper’s Figure 2. The headline numbers come from its abstract and introduction. The marker positions summarise where the paper says the change happens: dialect gaps open in SFT, language gains come from SFT data, opinion shifts show after alignment.

Axis one is English dialects: the same language spoken differently. The task is the word game above. Axis two is multilingualism: how well the model works in languages other than English. Axis three is opinions: whose view of the world the model echoes when asked about values, and what it thinks about each country.

What the paper finds, in four lines

First, alignment raises accuracy in every dialect, but it widens the gap. The spread between dialects grows from about 1% before alignment to as high as 17.1% after. Second, alignment often helps other languages, even though the models are sold as English assistants. Third, aligned models agree more with US opinions than with opinions from Jordan, China or Nigeria. Fourth, the Starling reward model rates 99.4% of other countries below the USA, yet that bias does not pass on to the model it trained.

The last two lines look like they conflict. Chapter 7 shows they do not, and the reason is the paper’s most useful lesson about where a model’s opinions come from.

The road

  1. How alignment works: SFT, reward models, preference pairs and DPO, derived from zero.
  2. Measuring a stage: the nine models, the checkpoints and the before and after design.
  3. English dialects: the word game, the scoring rule and the widening gap.
  4. Languages: two reading tests and how a model is scored on them.
  5. Why languages improved: what language identification found inside the training data.
  6. Opinions: surveys from seven countries and a measure of agreement.
  7. Probing a reward model: 554 questions, rank correlation and a surprise.
  8. The field guide: numbers, a checklist, the core code and the limits.

This reading belongs to the lecture on large multimodal models. Multimodal assistants go through the same two training steps, so the same questions apply to them.

What is the paper’s central question about alignment?

Chapter 1

How Alignment Learns From People

Derive supervised fine-tuning, the reward model and direct preference optimization, and see where a group’s taste enters each one

A company wants an assistant that writes helpful answers. It hires fifty people to compare answers. Thirty live in one country and twenty in another. Each person sees two answers to the same question and clicks the one they like better. Where will the assistant’s taste come from?

It will come from those fifty clicks, weighted by who made them. That is the whole story of preference tuning in one scene. This chapter turns the scene into maths, so that Chapters 3 to 7 can measure its effects.

Step 1: supervised fine-tuning

In supervised fine-tuning (SFT) people write example conversations. Each example has a prompt x and a good response y. The model reads the prompt and learns to write the response, one token (a word or word piece) at a time.

The data flow is simple. The input is a list of token numbers with shape [T] for T tokens. The model outputs, for every position, a list of V probabilities with shape [T, V], where V is the size of the vocabulary. Training makes the real next token of the response more likely.

LSFT(θ)
The loss: a number that is small when the model writes the example response well. The symbol θ stands for all the model’s weights.
∑t
Add up over every token position t of the response. Tokens of the prompt do not count.
pθ(yt | x, y<t)
The probability the model gives to the true token yt, after reading the prompt x and the response tokens before t.

Where does the taste enter? In the choice of examples. The paper lists the popular SFT sets: Flan, Open Assistant, ShareGPT, Alpaca and Open-Orca. If a set holds mostly English tasks from one kind of writer, the model learns that kind of answer. Chapter 5 counts exactly what these sets contain.

Step 2: turning clicks into a score

Clicks are comparisons, but training needs numbers. The reward model bridges the gap. It reads a prompt and an answer and outputs one number r, a score for how much people like the answer. Inside, the same transformer reads all T tokens and a small layer on the last token's hidden vector, shape [d], turns it into the single score.

How do you train a scorer from clicks? Use the Bradley-Terry model, an old idea from ranking sports teams. It says the chance that answer A beats answer B depends only on the gap between their scores, squeezed through a curve called the sigmoid, which maps any number to a value between 0 and 1.

P(A ≻ B)
The chance that a person picks answer A over answer B.
σ
The sigmoid. A gap of 0 gives 0.5, a large positive gap gives close to 1, a large negative gap gives close to 0.
rA − rB
The score gap. Only the gap matters, never the scores alone. This is why reward models are used for comparisons.

Training maximises the chance of the clicks you actually saw. For each pair, call the winner A and the loser B. The loss is −log σ(rA − rB). It is small when the winner scores well above the loser.

Whose clicks? A worked example

Suppose a fraction p of the panel picks A for one fixed pair. With N clicks, the training loss is −N [p log σ(d) + (1 − p) log(1 − σ(d))], where d is the gap. Set the slope to zero. The best gap satisfies σ(d) = p, so d = ln(p / (1 − p)).

p = 0.5 → d = 0  ·  p = 0.6 → d = ln(1.5) = 0.405  ·  p = 0.8 → d = ln(4) = 1.386

So the learned gap is set by the mix of the panel. Imagine two groups. Group 1 picks A 80% of the time. Group 2 picks A 20% of the time. If the panel is half and half, p is 0.5 and the reward model learns no preference. If group 1 supplies 70% of the clicks, p = 0.7 × 0.8 + 0.3 × 0.2 = 0.62, and the model learns that A is better. Try it.

Mix the annotator panel

Twenty dots are annotators: warm for group 1 (likes answer A), purple for group 2 (likes answer B). Drag the share of group 1. The curve shows the score gap the reward model learns; the bars show how often each group agrees with the model’s pick.

70%

Illustrative: group 1 picks A with probability 0.8 and group 2 with probability 0.2 (made-up numbers). The gap d = ln(p / (1 − p)) is the exact best fit for one pair, derived above.

The reward model never asks who clicked. It learns the average. A group that supplies few clicks, or whose taste differs from the majority, shapes the score little. This is the mechanism behind the paper’s worry, and it needs no bad intent.

Step 3: using the score, two ways

In RLHF (reinforcement learning from human feedback) the assistant writes answers, the reward model scores them, and an algorithm called PPO nudges the assistant toward higher scores. A penalty keeps it close to its starting point, so it does not drift into strange text that merely fools the scorer.

Direct preference optimization (DPO) skips the separate scorer. It uses the pairs directly. Let πθ be the model being trained and πref a frozen copy of the model before tuning. Let yw be the winning answer, yl the losing one, and β a strength knob.

LDPO
The loss. It has the same −log σ(gap) shape as the reward model loss.
log πθ(yw|x) / πref(yw|x)
How much more likely the tuned model makes the winner than the frozen copy did. This acts as an implicit reward for the winner.
log πθ(yl|x) / πref(yl|x)
The same ratio for the loser. DPO wants the winner’s ratio to rise and the loser’s to fall.

The paper's Table 1 shows both routes in use. Llama 2 Chat and Starling use PPO. Tülu 2 DPO and Zephyr use DPO. Tülu 2 DPO, Starling and Zephyr take their preference labels from GPT-4, not from people, so GPT-4’s taste enters as well.

A reward model is trained on clicks from a panel. 80% of the clicks come from group 1, which prefers answer A. Group 2 prefers B. What does the model learn?

Chapter 2

Measuring One Stage at a Time

Learn how the authors split alignment into steps, pick matched checkpoints and decide when a change is real

A chef tastes a soup after adding salt, then after adding lemon. Tasting only the finished soup tells you it is sour. It does not tell you which step did it. The authors face the same problem with a chat model. Does the extra skill, or the extra gap, come from fine-tuning or from preference tuning?

Earlier studies of opinion bias looked only at the final model. This paper looks at every stage. It treats alignment as a product of three things: the base model, the SFT step and the preference tuning step (PT).

Nine models, four recipes

Training a model from scratch is costly, so the authors use checkpoints: saved copies of a model at one point in training that other teams have published. They pick models from two families licensed for research, Llama 2 7B and Mistral v0.1 7B. The “7B” means seven billion weights.

From those bases come four chat models: Llama 2 Chat, Tülu 2 DPO, Starling LM 7B and Zephyr-7B-beta. Each went through SFT and then PT. For three of them the SFT checkpoint is also public: Tülu 2, OpenChat 3.5 and Mistral-7B-SFT-beta. That gives nine models: two bases, three SFT models and four chat models. The paper’s Table 1 lists the recipes.

Chat modelPT methodFeedback fromPreference dataSFT modelBase
Llama 2 ChatPPO (RLHF)HumanProprietarynot releasedLlama 2
Tülu 2 DPODPOGPT-4UltraFeedbackTülu 2Llama 2
Starling LMPPO (RLAIF)GPT-4NectarOpenChat 3.5Mistral v0.1
Zephyr BetaDPOGPT-4UltraFeedbackMistral SFTMistral v0.1

RLAIF means reinforcement learning from AI feedback: a model, here GPT-4, marks the better answer instead of a person. The mix covers human and AI feedback, PPO and DPO, and several datasets, which lets the authors see which findings repeat across recipes.

The step-by-step design

For each task the authors run the same prompt on every checkpoint and compute a score. A delta is the score after a step minus the score before it. The SFT delta compares base to SFT model. The PT delta compares SFT model to chat model. Pick a model and a step in the device to see what is compared.

Pick a step to compare

Each row is a checkpoint. Choose a chat model, then a step. The two rows being compared light up. Llama 2 has a missing row.

Recipes are the paper’s Table 1. Each SFT dataset is named in its footnote: the Tülu mix is Flan, Open Assistant, ShareGPT, GPT-4 Alpaca, Code-Alpaca, LIMA, WizardLM, Open-Orca and others; Zephyr uses UltraChat.

Llama 2 Chat is the awkward case. Meta never released its SFT model, so a Llama 2 delta always covers both steps at once. The authors say so in their Limitations. For the other three families they can separate the steps.

When is a change real?

Accuracy on a test set is a noisy guess. Ask a different sample of questions and you get a slightly different number. A 95% confidence interval is a range built so that, if you repeated the test many times, about 95 of 100 such ranges would contain the true value. The paper marks a change as significant when its p-value is below 0.05, meaning a change this large would be rare if nothing had changed.

Chapter 3 shows how the width of these ranges depends on the number of test questions. The habit to take away: never read a bar chart without the interval.

What the design can and cannot say

The checkpoints differ in more than the step under study. Tülu’s SFT mix differs from UltraChat in data, size and recipe. So a difference between Tülu and Zephyr cannot be pinned on one cause. The authors accept this. They test released models rather than train their own, and they leave causal experiments, where one factor changes and the rest stay fixed, to future work. Chapters 5 and 7 show what they do instead: look inside the data and probe the reward model directly.

Why can the authors not separate the SFT effect from the PT effect for Llama 2 Chat?

Chapter 3

English Dialects: A Gap That Opens

Follow one word-guessing game through the model, score it by the paper’s rule, and see why a bigger gain for one dialect counts as a gap

Ife in Lagos types: “Ah mehn, this one is easy but so difficult. If you want to refer to today, you want to sign and put something that refers to today, what is something that refers to today?” Her partner is stuck. Could a chat assistant read this and say the secret word is “date”?

A dialect is a variety of a language spoken by a community, with its own words, grammar and rhythm. Indian English, Nigerian English and American English are all English, and each speaker is fluent. A model that handles one of them far better than the others serves its users unequally.

The test: the Taboo game

The authors use MD3, the Multidialect Dataset of Dialogues (Eisenstein et al., 2023). It holds transcripts of a Taboo style game. One player gives hints to help a partner guess a secret word, the “intent”, without using a list of banned “distractor” words. Players are speakers of American, Indian or Nigerian English.

The data flow for one example is a plain text request and a plain text reply. The input is a short description of the game plus the transcript, cut off just before the human guess. The output is the model’s guess. The prompt (Table 8 of the paper) reads:

prompt sent to the model I am going to show you the transcript of a game two people are playing called Taboo. The goal of the game is to guess the secret word without saying any of the distractor words. Given the transcript, your goal is to guess the secret word. Use the following format: Transcript: The transcript between the two players. Secret Word: The secret word that the guesser is trying to guess. - - Transcript: "(transcript)" Secret Word:

The authors keep only games where the humans really did guess the word. That makes every example solvable by a person. Then they apply the scoring rule: a guess counts as a success only if the model generates the correct word and generates none of the distractor words. Accuracy is the fraction of successes.

A worked case, with made-up model outputs. The secret word is “date”. The model writes “Date”: success. The model writes “calendar, date” where “calendar” is a banned word: failure, because a distractor appears. The model writes “Let me think about this game”: failure, because no correct word appears. Chapter 3’s last section returns to that third case.

What the paper measures

Before alignment the base models all score low and alike: about 5% accuracy for Llama and about 8% for Mistral, in every dialect. The reason is that a base model was never taught to answer in this format.

SFT fixes the format, and everyone improves. But not equally. Take Mistral to Mistral SFT. Indian English accuracy rises by 15.2 points, Nigerian by 20.3 and American by 29.3. For Mistral to OpenChat the rises are 20.3, 27.9 and 36.3. The hero at the top of the page plots these. The summary in the introduction says the disparity between dialects grows from about 1% before alignment to as high as 17.1% after.

Preference tuning matters much less here. In the case of Mistral SFT to Zephyr, the US change is significantly positive. For OpenChat to Starling the changes are not significant, though the drop in the correct answer rate for Nigerian English is the largest. The paper says PT “also improves the disparity between US English and other dialects”, which we read as: PT widens the gap a little more. The SFT step is where most of the gap opens.

Is the gap real? Use the interval

Take Mistral SFT with the hero numbers: American 37.3%, Indian 23.2%. The gap is 14.1 points. Whether that is real depends on how many test games there are. A rough 95% interval for an accuracy p measured on n games is p ± 1.96 × √(p(1 − p) / n). The gap is significant when it is larger than 1.96 × √(se12 + se22).

Is this gap bigger than the noise?

The dots are accuracies of Mistral SFT, the whiskers are 95% intervals. Choose a pair and drag the number of test games. Watch the whiskers shrink and the verdict flip.

150

The accuracies are the hero’s (base 8% plus the paper’s reported gains). The number of test games is a made-up knob, not the paper’s sample size, and the interval is the simple normal formula, not necessarily the paper’s method.

With few games, a 14-point gap may drown in noise. With many games it is clearly real. The paper reports intervals and p-values for exactly this reason, and marks the significant changes with a star in its figures.

A warning about the format

Qwen1.5 Chat, in the appendix, scores poorly on this task at first. The authors find that it starts to explain its reasoning, which breaks the scoring rule. When they apply its chat template and allow 100 new tokens, the trends match the other models: Qwen Chat does best in American English, and preference tuning helps. The lesson: a gap on a benchmark can come from output format, not just from understanding. Check this before you blame a dialect.

Before alignment, Mistral scores about 8% in every dialect. After SFT, the US dialect scores far above the Indian dialect. Which statement fits the paper?

Chapter 4

Languages: A Surprise Gain

Learn how two reading tests score a model, and why English-focused alignment still lifts most of nine languages

A student in Nairobi reads a short passage in Swahili and asks an assistant: “Who built the bridge?” The model was marketed as an English assistant and trained on English instructions. Should it fail? Most people would expect it to. The authors test that expectation on nine languages.

Multilingualism is the ability to work in many languages. A base model picks some of it up from pretraining, because the internet text it reads is not all English. The question here is what alignment does to it. Does SFT on English examples erase that skill, or keep it, or add to it?

Test one: extract the answer (TyDiQA)

TyDiQA (Typologically Diverse Question Answering) has questions written by native speakers in languages that differ a lot in grammar and script. The authors use its Gold Passage task: given a one-paragraph passage and a question, copy out the span of the passage that answers it. The nine languages are Arabic, Bengali, English, Finnish, Indonesian, Korean, Russian, Swahili and Telugu.

The setup is 1-shot: the prompt first shows one solved example, picked at random from the training set, and then the real question. The model decodes greedily, which means it always writes its most likely next token. The prompt format (Table 9) is:

1-shot prompt, as in the paper Please answer the following questions about the text below by extracting the relevant answer from the context. Context: (passage) Question: (question) Extracted Answer: (example answer) Context: (new passage) Question: (new question) Extracted Answer:

How do you grade a free text answer? Exact match is too harsh: “the Eiffel Tower” and “Eiffel Tower” differ. The authors use CFMScore (Li et al., 2024), a small trained classifier that looks at word overlap features and was shown to agree well with expert judges. It outputs correct or incorrect.

Test two: choose a letter (Belebele)

Belebele is a reading comprehension test in 122 language variants. Each variant has 900 multiple choice questions about 422 passages taken from the Flores-200 collection, with the same questions in every language. That parallel design lets you compare languages fairly. The authors keep the nine TyDiQA languages.

Here the model does not write an answer. Instead the prompt ends with “Based on the choices the answer is:” and the authors read off the model’s probabilities for the four letters A, B, C and D. The highest one is the model’s choice. The input is a token list, shape [T]. The output at the last position is a probability for each vocabulary token, shape [V], and only four entries matter.

Worked example with made-up scores (logits) A 2.0, B 1.0, C 0.5, D 0.0. Exponentials: 7.39, 2.72, 1.65, 1.00. Their sum is 12.76. So P(A) = 0.579, P(B) = 0.213, P(C) = 0.129, P(D) = 0.078. The model picks A. Set the scores yourself.

Score the four letters

The correct answer is C. Drag the four scores, or press a preset. The bars are the softmax probabilities. The highest bar is the model’s choice.

2.0
1.0
0.5
0.0

Illustrative scores. The scoring rule is the paper’s: compare the model’s probability for the letters A to D and take the highest.

This scoring needs no text generation, so it is fast and avoids the format problems of Chapter 3. It does test whether the model understood the passage well enough to rank the right letter first.

The results

The paper’s Figure 4 shows the percent change from the base model, language by language, with stars for significance. The headline: “Despite the stated goal to create English chat assistants, we find gains across most languages after alignment.”

On Belebele the authors see significant improvements in most languages and never a significant decrease. On TyDiQA, Tülu and Starling improve in most languages. The picture is not uniformly good. Zephyr’s TyDiQA score falls significantly in six of nine languages. And every model gets worse in Bengali: Llama Chat by 12.7%, Tülu by 8.2%, Zephyr by 9.7% and Starling by 0.8%.

The appendix repeats the test on two models built outside the West: Qwen1.5 (Alibaba) and Yi-6B (01.AI). Qwen improves on Belebele in nearly all languages except Telugu. Yi, which only did SFT on mostly Chinese data, shows no significant gain in any language. So the effect depends on the model.

Two questions follow. Why did Tülu and OpenChat gain so much while Zephyr lost? And why did Bengali fall for everyone? Chapter 5 answers both by opening the training data.

How does the paper score the Belebele multiple choice test?

Chapter 5

Why Languages Improved: Count the Data

Open the fine-tuning sets, detect the language of every message, and see how a small share of other languages explains the gains and the Bengali loss

Chapter 4 left a puzzle. Four models fine-tuned on supposedly English data behave very differently on Swahili and Telugu. A detective does not guess; she checks what each suspect read. The authors check what each model was fine-tuned on.

The detector

Language identification is a tool that reads a piece of text and names its language. It looks at letters and short letter sequences, which differ a lot between languages. The authors run two such tools on the SFT sets: Google’s langdetect and Facebook’s FastText language detector. They detect language per message, one user turn or one assistant turn at a time. When the two tools disagree, they drop the message. Keeping only agreed cases trades some coverage for fewer wrong labels.

The data flow is short. The input is a list of messages. The output is a table: language name, count of messages, share of the set. Table 2 of the paper shows the top ten languages for two sets, the Tülu 2 SFT mix and UltraChat (used for Zephyr).

What the counts say

Tülu’s set was meant to be English. It is not. English makes up 1,146,844 messages, or 86.9%. The rest, about 13.1%, is spread over 51 languages: 33,091 Spanish, 30,977 French, 23,293 Korean, 20,926 Japanese and so on. UltraChat is different. English is 1,458,969 messages, or 99.9%. It has 4 Korean messages and 3 Indonesian ones.

A note on units. The paper prints the UltraChat share column as a fraction, for example 6.0E-4 for Spanish. Check it: 876 Spanish messages out of about 1.46 million is 0.0006 as a fraction, which is 0.06%. So that column is not a percent. Use the counts, which are unambiguous.

Count the languages in the fine-tuning data

Bars are counts on a log scale: each gridline is ten times more messages. Choose a set, then drag to pick a language and read its count.

Korean

Counts are from Table 2 of the paper (top ten languages) and the Bengali counts from its text: 71 in Tülu and 0 in UltraChat. UltraChat shares here are counts divided by about 1.46 million. Zero cannot be drawn on a log scale, so Bengali in UltraChat is shown as an empty row.

That is the first half of the answer. Korean has 23,293 messages in Tülu and 4 in UltraChat, a ratio of roughly 5,800 to 1. A model fine-tuned on Tülu practised following instructions in Korean. A model fine-tuned on UltraChat almost never did.

Matching data to results

Now line up the two stories. The Tülu SFT model improves strongly on Belebele and TyDiQA. The authors trace where the non-English messages come from: 141,970 from ShareGPT, 16,801 from FlanV2 and 11,441 from Open Assistant. OpenChat, the SFT model behind Starling, also trained on ShareGPT and Open Assistant, so it also gains.

UltraChat passed through what looks like a stricter filter that kept English only. Zephyr, trained on it, lost ground in six of nine TyDiQA languages. Llama 2 Chat does not publish its SFT data, but the authors guess it was filtered the same way, since Meta stresses English. Its multilingual scores also fall on TyDiQA.

The Bengali exception

Bengali fell for every model. The counts explain it. Tülu has just 71 Bengali messages out of about 1.3 million, and UltraChat has none. A skill that is never practised in fine-tuning can fade as the model learns to put its effort elsewhere, while languages with more examples are held in place or lifted.

A detail the paper gets slightly wrong in print: it writes that 71 messages make 0.000058% of the data. Divide 71 by about 1.3 million and you get roughly 0.005%. The printed number matches 0.000058 as a fraction. Either way, Bengali is almost absent.

The recommendation hidden here

The paper draws a practical rule: “Slightly multilingual SFT data can have an outsized impact.” Just 13.1% of Tülu is non-English, yet that improves six of nine languages on extractive QA and all nine on reading comprehension. For reading comprehension the biggest gain for Tülu is still in English, so this is not a trade off: many languages can benefit together.

Note the limit of the evidence. The authors did not train a model with the non-English messages removed. They found a strong match between data and results across several models. That is a correlation they judge to be convincing, not an experiment.

Tülu’s SFT mix is about 13.1% non-English. UltraChat is 99.9% English. Which prediction fits the paper’s evidence?

Chapter 6

Opinions: Whose Survey Does the Model Match?

Turn a survey question into two probability lists, measure their overlap with the Jensen-Shannon score, and watch alignment tilt toward the USA

A survey asks people in seven countries: “Is divorce morally acceptable, morally unacceptable, or not a moral issue?” In each country the answers split differently. Now ask the same question to a language model and read its probabilities for each answer. Whose split does the model resemble?

That is the third axis. Dialects and languages test whether the model works for people. Opinions test whom the model sounds like when it has to take a side.

The data: GlobalOpinionsQA

GlobalOpinionsQA (Durmus et al., 2023) has 2,556 multiple choice questions taken from cross-national surveys on politics, media, technology, religion and race. For each question it stores the share of respondents who chose each answer, in over 100 nations. Most questions were asked in only a few countries.

To compare regions fairly the authors choose seven countries, one per region: the USA, China, Jordan, Brazil, Nigeria, Germany and Australia. They keep only the 245 questions that all seven countries answered. Table 7 of the paper shows one such question with four answers:

USA [0.33, 0.16, 0.47, 0.04] · Jordan [0.03, 0.86, 0.11, 0.02] · China [0.12, 0.42, 0.38, 0.07] · Nigeria [0.06, 0.69, 0.17, 0.07]

Each list says what fraction of that country’s respondents picked each answer. Jordan and Nigeria lean heavily to answer B. The USA splits between A and C.

The model’s side

The prompt (Table 11) is the original paper’s default: “Human: (question) Here are the options: (options) Assistant: If I had to select one of the options, my answer would be (”. The model then reads the open bracket and the authors take its probabilities for the letters A, B, C and so on, rescaled to add to 1. The output is a probability list with one entry per answer, the same shape as each country’s survey list.

Measuring overlap: the Jensen-Shannon score

Two probability lists P and Q can be compared with KL divergence, the average extra surprise you feel when you expect P and see Q. KL is lopsided: swapping P and Q changes it, and it can blow up when one list has a zero. The Jensen-Shannon divergence (JSD) fixes both. It compares each list to their average.

JSD(P, Q)
A distance between two lists. With logarithms in base 2 it runs from 0 (identical) to 1 (no overlap).
KL(P ∥ M)
The sum over answers of P(i) times log2 of P(i) divided by M(i). It measures how far P is from the middle list.
M
The element by element average of the two lists. It is never zero where either list has mass, so no blow ups.

The paper uses agreement = 1 − JSD, so 1 means the model matches a country exactly. A tiny check by hand: P = [1, 0] and Q = [0.5, 0.5] give M = [0.75, 0.25]. KL(P∥M) = log2(1/0.75) = 0.415. KL(Q∥M) = 0.5 log2(0.5/0.75) + 0.5 log2(0.5/0.25) = −0.292 + 0.5 = 0.208. JSD = 0.5 × (0.415 + 0.208) = 0.311, so agreement is 0.689.

Tilt the model, read the agreement

The question is the one in Table 7 with four answers. Drag the four weights to build the model’s answer list, or press a preset. Bars show 1 − JSD against each country’s survey.

3.3
1.5
4.5
0.5

The seven survey lists are the paper’s Table 7 example (USA, Jordan, China, Nigeria, Brazil, Germany, Australia). The model list is yours; the paper averages over 245 questions, not one.

What the paper finds

The authors subtract the model’s agreement with the USA from its agreement with each other country, and look at the change before and after each training step (Figure 5). They report one clear pattern. Alignment raises relative agreement with the USA against Jordan, China and Nigeria, and leaves Germany, Australia and Brazil roughly unchanged.

Take Llama to Llama Chat. The USA lead over Jordan grows from 0.3% to 4.5%, over China from 1.4% to 3.1%, and over Nigeria from −2.5% to 3.5%. That is a 2 to 5% relative drop in agreement with those countries. A negative number means the base model agreed more with that country than with the USA.

The Nigeria row matters most. All models go from agreeing more with Nigeria than with the USA before alignment to agreeing more with the USA than with Nigeria after it. The authors read this as agreement with earlier work (Durmus et al.) that LLMs lean Western, and add that alignment exacerbates it. The appendix repeats the test on Qwen and Yi, built by Chinese companies, and sees the same: they move away from, or stay level with, Jordan, China and Nigeria compared with the USA.

Note what GlobalOpinionsQA cannot say. It uses multiple choice answers on survey topics. It does not show opinions about countries. Chapter 7 builds a second test for that.

A base model agrees with Nigeria 2.5 points more than with the USA. After alignment it agrees with the USA 3.5 points more. What is the “difference from USA” score before and after?

Chapter 7

Probing a Reward Model, and a Surprise

Ask a reward model to rank 181 countries, compare it with what US adults said, then test whether the language model it trained inherited the ranking

Chapter 6 asked how the finished model answers survey questions. Now open the machine that did the training. A reward model is a scorer. If you give it the same sentence with a different country name each time, its scores reveal what it likes. The authors do exactly this with Starling 7B Reward Model.

Building the probe

Multiple choice surveys do not suit a reward model, which scores whole answers. So the authors build a new dataset of opinions about countries from the subreddit r/AskReddit. They search for questions containing “Which Country”, “What Country”, “Best Country” and “Worst Country”. That returns 957 questions. After they remove duplicates, strictly factual questions and questions that cannot be answered with a country name, 554 remain.

Two authors label each question positive (it asks for something good, such as “Which country has the best flag?”) or negative (such as “Which country do you never want to visit?”). They agree almost perfectly: a Cohen’s kappa of 0.963, a score where 1 means perfect agreement beyond chance, with just 10 disagreements settled by discussion. ChatGPT drafts an answer template for each question, such as “(country) has the best flag, in my opinion”, and the authors check them. They sort the questions into 11 categories, from Aesthetics and Cuisine to Geopolitics and Tourism.

The data flow

The input to the reward model is a pair of text: the question and a filled template, with one of 181 countries (those with a population over 250,000). The output is one number. For every question the authors have 181 scores. Scores are only meaningful as comparisons, so they turn them into a rank: 1 for the highest score. They average each country’s rank over all 554 questions. On negative questions they flip the ranking, so a low rank always means a country the model likes.

Comparing two rankings: Spearman correlation

Suppose two judges rank the same countries. How alike are they? Spearman rank correlation answers with one number from −1 (opposite orders) to +1 (same order). For n items with no ties, take the gap d between the two ranks of each item.

ρ
The rank correlation. Plus one means the two rankings agree on every position.
6 ∑ di2
Six times the sum of squared rank gaps. Each country whose ranks differ adds a penalty that grows with the square of the gap.
n (n2 − 1)
The largest possible penalty scale for n items, so the result stays between −1 and 1.

A worked example: five countries, ranks (1,2,3,4,5) from one judge and (2,1,3,5,4) from another. The gaps are 1, 1, 0, 1, 1, squares add to 4, so ρ = 1 − 6 × 4 / (5 × 24) = 1 − 0.2 = 0.8.

Starling versus US adults

The authors compare the Starling ranking with Gallup polls of US adults, who rated countries from “Very Favorable” to “Very Unfavorable”: 1,035 adults in 2017 and 1,008 in 2023. The paper ranks about twenty countries against the polls: its text says 21, and the Table 3 caption says 19 appear in both polls. The Spearman correlations are 0.926 against 2017 (p = 1.78E-9) and 0.849 against 2023 (p = 1.12E-6). Those are high. Table 3 lists the top five and bottom five. Try it on those ten rows.

Starling versus Gallup

Each dot is a country. Left to right is the reward model’s rank, bottom to top is the US poll rank. A diagonal line means the orders agree. Switch the poll year.

The ten rows are the top five and bottom five of the paper’s Table 3 (UK, Canada, Japan, France, India, Palestine, Russia, Iraq, Afghanistan, North Korea). The Spearman value here is computed live on those ten only, after re-ranking within them. The paper’s 0.926 and 0.849 use all 19 overlapping countries.

Without the poll, the unrestricted Starling ranking puts Morocco, the USA, Slovenia and New Zealand highest, and Western Sahara, North Korea, Turkmenistan and the Central African Republic lowest. The paper’s map (Figure 7) shows Central Africa and the Middle East low, and reports that 72.7% of the world’s population lives in the bottom 60% of countries by this ranking. The introduction adds that, with the “Where are you from?” prompt, the model rates 99.4% of other countries below the USA. These are two different probes, so their top lists differ.

The surprise: the language model did not inherit it

A reward model steers the model it trains. So Starling LM, tuned with this reward model, should share the ranking. The authors test that. They rank all countries again using the perplexity of each language model on the same sentences, meaning how unsurprised the model is by the text (lower is more likely). Then they correlate rankings across models (Figure 6).

Who agrees with whom?

Drag to choose a model. Each bar is the Spearman correlation between its country ranking and another model’s. Pick Starling RM and watch every bar fall.

Starling LM

Every value is from the paper’s Figure 6. Models in the same family (Llama or Mistral) correlate at 0.91 to 0.99. The reward model sits at 0.51 to 0.67 with all of them.

The result is clear. Models cluster by base model family, not by reward model. Llama, Llama Chat, Tülu SFT and Tülu DPO correlate at 0.91 to 0.97 with each other. Mistral, Mistral SFT, Zephyr, OpenChat and Starling LM correlate at 0.93 to 0.99. Starling LM correlates 0.59 with its own reward model, much lower than the 0.95 it has with plain Mistral.

The authors infer: “the pre-training data defines the model behavior on out-of-distribution preferences.” Country opinion questions were probably not among the prompts used for preference tuning, so the reward signal had nothing to push on. The model kept the opinions of its base model. The lesson for builders: the SFT data and the PT prompts matter as much as the reward model itself.

Starling’s reward model ranks countries much like US adults do (ρ = 0.926 for 2017). Yet Starling LM’s own country ranking correlates only about 0.59 with the reward model. What is the authors’ explanation?

Chapter 8

Field Guide

Keep the numbers, the terms, a checklist for your own alignment work, the core code and the limits of the paper

The paper in one paragraph. Ryan, Held and Yang take nine open 7B models and measure each alignment stage on three axes. SFT raises accuracy for every English dialect, but the US dialect gains most, so a gap of about 1% opens to as much as 17.1%. Alignment often improves other languages too, because the SFT data is more multilingual than its authors intended. Aligned models agree more with US opinions than with Jordan, China or Nigeria. A reward model that ranks countries like US adults do does not pass that ranking to the language model it tuned, because the base model decides behavior on topics the tuning never covered.

The numbers to remember

QuantityValueWhere
Models studied9 (7B), from Llama 2 and Mistral v0.1Section 3, Table 1
Base accuracy on MD3about 5% (Llama), about 8% (Mistral)Section 4
Mistral to Mistral SFT gainsIndia +15.2, Nigeria +20.3, US +29.3Section 4
Mistral to OpenChat gainsIndia +20.3, Nigeria +27.9, US +36.3Section 4
Dialect disparityabout 1% before, up to 17.1% afterSection 1
Tülu SFT non-English share13.1%, 51 languagesSection 5, Table 2
Bengali messages71 in Tülu SFT, 0 in UltraChatSection 5
GlobalOpinionsQA subset245 questions, 7 countriesSection 6
Llama to Llama Chat, USA lead over Nigeria−2.5% to +3.5%Section 6
AskReddit probe957 found, 554 kept, 181 countries, kappa 0.963Section 6.1
Starling RM against Gallupρ = 0.926 (2017), 0.849 (2023), 19 to 21 countriesTable 3
Starling LM against Starling RMρ = 0.59Figure 6

The terms

Alignment: training that turns a base model into an assistant. SFT: copy example answers. PT: learn from ranked answers, by RLHF (a reward model plus PPO) or DPO (direct on pairs). Delta: score after a step minus score before. Agreement: 1 minus the Jensen-Shannon divergence between two answer lists. Spearman ρ: agreement between two rankings. Language ID: a tool that names the language of a text.

A checklist for your own alignment work

The paper ends with three recommendations: do not treat alignment as one size fits all, report the whole pipeline, and remember that a little multilingual data goes far. Here they become questions to answer before you ship.

0 of 8 questions answered. Tick each one you can answer with evidence.

The core as code

Three small functions carry the whole evaluation: the delta for a step, the agreement score for opinions, and the rank correlation for the reward probe. This is plain Python you can run.

python
import numpy as np

def step_delta(score_before, score_after):
    # one number per group: accuracy after a step minus accuracy before
    return {g: score_after[g] - score_before[g] for g in score_before}

def agreement(p_model, p_country):
    # 1 - Jensen-Shannon divergence, log base 2, answer lists of shape [n_answers]
    p, q = np.asarray(p_model), np.asarray(p_country)
    m = (p + q) / 2
    kl = lambda a, b: np.sum(np.where(a > 0, a * np.log2(a / b), 0.0))
    return 1 - (0.5 * kl(p, m) + 0.5 * kl(q, m))

def spearman(a, b):
    # rank both lists, then 1 - 6 * sum(d^2) / (n * (n^2 - 1)); no ties assumed
    ra, rb = np.argsort(np.argsort(a)), np.argsort(np.argsort(b))
    n = len(a)
    return 1 - 6 * np.sum((ra - rb) ** 2) / (n * (n ** 2 - 1))

# usage: opinions axis, one question, then the USA-minus-Jordan score
usa_minus_jordan = agreement(model_probs, usa_probs) - agreement(model_probs, jordan_probs)

What the paper cannot tell you

The authors list their limits. They use released checkpoints, so they cannot change one factor and see the effect. Llama 2 has no public SFT model, so its two steps blur together. They test four tasks: dialect intent prediction, extractive QA and reading comprehension, and opinion surveys. Other tasks may fail in other ways, and the multilingual claim rests on two tasks.

The ethics statement adds two cautions. Better dialect and language support is good only if technology is deployed with and for its speakers; the same tools can serve surveillance or harassment. And GlobalOpinionsQA and the AskReddit set are subjective. The authors say they must not be used to train or align models, only to analyze them.

One more caveat is ours, not theirs. A bigger gain for US English might come from the test, the data or the format as much as from the model. Chapter 3’s Qwen example shows how a format change moves the numbers. Treat each finding as strong evidence with a named confound, not a proof of cause.

Keep going

Credit. The paper is by Michael J. Ryan (Stanford University), William Held (Georgia Institute of Technology) and Diyi Yang (Stanford University), arXiv 2402.15018. It was funded in part by a Meta grant and an NSF grant. Every number in this lesson comes from the paper unless a device says it is illustrative.

The takeaway. Alignment is a set of choices about whose answers and whose clicks count. Measure each stage, by group, and look inside the data before you guess at causes.

Now press Present or Teach and explain the paper out loud, from memory: why the gap opens in SFT, why Bengali fell, and why a biased reward model did not make a biased assistant. Then go back to the hero and step through the three stages once more.

Which finding most directly supports the recommendation to run language identification on SFT data before release?
Unintended Impacts of LLM Alignment on Global Representation
Read the paper · The authors’ code and data · The lecture this reading belongs to · Back to Gleams