Beta Distribution & CLT

Chapter 1: The Beta Distribution

The Beta distribution is a continuous distribution on [0, 1] controlled by two shape parameters, a and b. Its PDF is:

where B(a, b) = ∫₀¹ t^a−1(1 − t)^b−1 dt is the Beta function, a normalizing constant that ensures the PDF integrates to 1. You never need to compute B(a, b) by hand — every programming language has it built in.

The key statistics:

The parameters a and b control the shape in an intuitive way. Think of a as counting "successes + 1" and b as "failures + 1". When a > b, the distribution leans toward 1 (high probability). When b > a, it leans toward 0. When a = b, it is symmetric around 0.5.

Worked example — coin flips: You flip a coin 10 times and observe 9 heads, 1 tail. With a uniform prior (Beta(1,1)), your posterior belief about the coin's heads probability is X ~ Beta(1 + 9, 1 + 1) = Beta(10, 2). The mean is 10/12 ≈ 0.833 and the variance is 20 / (144 × 13) ≈ 0.0107, giving a standard deviation of about 0.103.

The Beta tells us: yes, the probability is probably high (the peak is near 0.9), but there is meaningful spread — values from 0.6 to 1.0 are all plausible. With 100 heads out of 110 flips, we would get Beta(101, 11), still centered near 0.9 but much tighter.

Chapter 2: Beta as Conjugate Prior

In Chapter 1, we started with a uniform prior Beta(1, 1) and updated it with observed data. But what if we already had some prior belief? Maybe we think the coin is roughly fair — we could start with Beta(5, 5) instead of Beta(1, 1). The beautiful result: the posterior is still a Beta.

Here is the derivation. Let our prior belief about the heads probability be X ~ Beta(a, b). After observing h heads and t tails:

The likelihood is Binomial: P(H=h, T=t | X=x) ∝ x^h(1 − x)^t. The prior is Beta: f(X=x) ∝ x^a−1(1 − x)^b−1. Multiplying:

That is the kernel of Beta(a + h, b + t). The normalizing constant handles itself. We just add the observed counts to the prior parameters.

Think of a and b as pseudo-observations: a − 1 imaginary successes and b − 1 imaginary failures. When you start with Beta(1, 1), there are 0 imaginary trials — complete ignorance. When you start with Beta(10, 10), you are asserting the equivalent of 18 coin flips that came out evenly. New real data gets blended with these pseudo-observations.

Worked example — strong prior vs. weak prior: You believe a coin is fair: prior Beta(10, 10). You then observe 8 heads, 2 tails. Your posterior is Beta(18, 12). The posterior mean is 18/30 = 0.60 — the prior has "pulled" your estimate toward 0.5. Had you used a weak prior Beta(1, 1), the posterior would be Beta(9, 3) with mean 0.75 — much more influenced by the data. The strength of the prior (measured by a + b) determines how many real observations it takes to overwhelm it.

Chapter 3: Adding Random Variables

We now shift to a completely different topic: what happens when you add two random variables together? If X and Y are random variables, what is the distribution of Z = X + Y? This is called convolution, and it underlies the Central Limit Theorem.

For discrete random variables, the PMF of the sum is:

If X and Y are independent, this simplifies to:

The idea: to get a sum of n, enumerate all the ways X and Y can combine to produce n. X could be 0 and Y could be n. X could be 1 and Y could be n − 1. Add up the probabilities of all these mutually exclusive cases.

Worked example — sum of two dice: Let X and Y each be uniform on {1, 2, 3, 4, 5, 6}. What is P(X + Y = 7)?

We need all pairs (i, 7 − i) where both i and 7 − i are in {1,...,6}. That gives i = 1,2,3,4,5,6 — all six values work. Each pair has probability (1/6)(1/6) = 1/36.

For P(X + Y = 4): pairs are (1,3), (2,2), (3,1). That is 3/36 = 1/12. The triangle-shaped PMF of the dice sum peaks at 7 because 7 has the most contributing pairs.

Chapter 4: Convolution Results

For several important distribution families, the sum of two independent random variables stays in the same family. These closed-form results save you from computing the convolution sum or integral by hand.

Proof sketch for Poissons: Let X ~ Poi(λ₁) and Y ~ Poi(λ₂). Using the convolution formula:

The last step uses the Binomial theorem: (a + b)ⁿ = ∑ C(n,k) a^k b^n−k. The result is exactly Poi(λ₁ + λ₂).

The Normal result is especially powerful because it extends to any number of independent summands. The means add, the variances add, and the result is still Normal. This is the algebraic backbone of the CLT.

Sum of two independent uniforms: If X, Y ~ Uni(0,1) independently, the PDF of Z = X + Y is triangular:

A uniform plus a uniform is not uniform. The distribution peaks at z = 1 and tapers linearly to zero at z = 0 and z = 2. Already with just two summands, we see the beginnings of a bell shape. This is the CLT in embryo.

Worked example — customer arrivals: Customers arrive at a store from two independent sources. Source 1: Poi(3) per hour. Source 2: Poi(5) per hour. Total arrivals per hour: Poi(3 + 5) = Poi(8). The probability of exactly 10 total arrivals: P(Z = 10) = e⁻⁸ · 8¹⁰ / 10! ≈ 0.0993.

Chapter 5: The Central Limit Theorem

Imagine summing 100 independent Uni(0, 1) random variables. Each one is flat and rectangular. But their sum? A beautiful bell curve, almost indistinguishable from a Normal. This is the Central Limit Theorem in action.

The two versions are equivalent: divide the sum by n and you get the average. The sum has variance nσ² (growing); the average has variance σ²/n (shrinking). Both are Normal.

What makes this remarkable is the word any. The original X_i can be uniform, exponential, Bernoulli, Poisson, or any other distribution — as long as it has finite mean and finite variance, the sum converges to Normal. The original distribution's shape gets washed out.

Worked example — 10 dice: Roll a 6-sided die 10 times. Let X = X₁ + ... + X₁₀. Each die has E[X_i] = 3.5 and Var(X_i) = 35/12. By the CLT, X ≈ N(10 × 3.5, 10 × 35/12) = N(35, 29.17).

What is P(X ≤ 25 or X ≥ 45)? Using a continuity correction (because dice are discrete):

Notice the continuity correction: we used 25.5 instead of 25 and 44.5 instead of 45. This accounts for the fact that we are approximating a discrete distribution with a continuous one. You need a continuity correction whenever the original X_i are discrete (dice, coins, Poisson counts). You do not need it when the X_i are already continuous (uniforms, exponentials).

Chapter 6: CLT Proof Sketch

Why does the CLT work? The full proof uses moment-generating functions (MGFs) or characteristic functions. Here is the idea at a high level.

Every random variable X has a moment-generating function M_X(t) = E[e^tX]. The key property: if X and Y are independent, then M_X+Y(t) = M_X(t) · M_Y(t). Multiplication in MGF-space corresponds to convolution in distribution-space.

Let Z_i = (X_i − μ) / σ be the standardized version of each X_i. The standardized sum is S_n = (Z₁ + ... + Z_n) / √n.

Now Taylor-expand M_Z(t / √n) around t = 0. Since Z is standardized, E[Z] = 0 and E[Z²] = 1:

Raising to the n-th power:

But e^t2/2 is exactly the MGF of the standard Normal N(0, 1). Since the MGF uniquely determines the distribution, the standardized sum converges to a standard Normal. Done.

Worked example — convergence rate: How fast does the CLT kick in? For symmetric distributions (like uniforms), n = 5 already gives a good Normal approximation. For heavily skewed distributions (like Exp(λ)), you might need n = 30–50. For Bernoulli(p) with p very close to 0 or 1, you might need n > 100. A common rule of thumb: np > 5 and n(1−p) > 5 for the Binomial.

Chapter 7: Showcase — CLT Demonstrator

This is the payoff simulation. Choose a source distribution (Dice, Uniform, Exponential, or Beta), set how many you sum (N), and watch the histogram of 2000 samples converge to a Normal as N increases. The orange curve is the theoretical Normal predicted by the CLT.

Worked example — algorithm runtime: You have an algorithm with unknown mean runtime μ and known variance σ² = 4 sec². You want to estimate μ to within ±0.5 seconds with 95% confidence. How many trials n do you need?

By the CLT, the sample mean X̄ ~ N(μ, 4/n). We need P(−0.5 ≤ X̄ − μ ≤ 0.5) = 0.95. Standardizing:

For 95% confidence, we need 0.5√n/2 = 1.96, so √n = 7.84, giving n = 61.4. We need 62 runs.

Chapter 8: Worked Problems

Problem 1: Beta expectation. A website A/B test shows 30 clicks out of 100 visitors for variant A. Using a uniform prior, what is the posterior distribution for the click-through rate, and what is its 95% credible interval?

With prior Beta(1, 1) and data h = 30, t = 70: posterior is Beta(31, 71). Mean = 31/102 ≈ 0.304. The 95% credible interval can be computed numerically: approximately [0.22, 0.40]. The posterior tells us the true rate is likely between 22% and 40%.

Problem 2: Convolution of dice. You roll two 4-sided dice (each uniform on {1,2,3,4}). What is P(sum = 5)?

Pairs that sum to 5: (1,4), (2,3), (3,2), (4,1). Each has probability (1/4)(1/4) = 1/16.

Problem 3: CLT for proportions. A fair coin is flipped 400 times. What is the probability of getting between 185 and 215 heads (inclusive)?

Let S = number of heads. S ~ Bin(400, 0.5). By CLT, S ≈ N(200, 100). Using continuity correction:

Problem 4: Updating with a strong prior. A doctor believes a treatment has a 60% success rate based on previous experience, modeled as Beta(12, 8). A new trial gives 15 successes and 25 failures. What is the posterior, and how does it differ from using a uniform prior?

With Beta(12, 8) prior: posterior = Beta(12 + 15, 8 + 25) = Beta(27, 33). Mean = 27/60 = 0.45.

With Beta(1, 1) prior: posterior = Beta(16, 26). Mean = 16/42 = 0.381.

The informative prior pulls the estimate up toward 0.6, while the data pulls it down toward 15/40 = 0.375. The compromise (0.45) is between the two. With more data, the prior's influence would diminish.

Problem 5: Sample size for CLT. Light bulbs have mean lifetime 1000 hours with SD 200 hours. An engineer tests n bulbs and averages their lifetimes. How large must n be so the average is within 20 hours of the true mean with 99% probability?

Chapter 9: Connections

The Beta distribution and the Central Limit Theorem are not isolated results — they connect to nearly every branch of statistics, machine learning, and engineering.

Master Beta and CLT and you hold the two keys to applied statistics: principled belief updating and the universal convergence to Normality.