Bayesian Distributions

Chapter 1: Gaussian Mixture Model

A single Gaussian can only model one bump. Real data often has multiple clusters — heights of men and women, income groups, cell types in a tissue sample. A Gaussian Mixture Model (GMM) is a weighted sum of K Gaussians, each representing one cluster.

Each component k has a weight π_k (how common that cluster is), a mean μ_k (cluster center), and a variance σ_k² (cluster spread). The weights must sum to 1: ∑π_k = 1.

python
import numpy as np
from scipy.stats import norm

def gmm_pdf(x, mus, sigmas, weights):
    """Evaluate GMM density at points x."""
    return sum(w * norm.pdf(x, m, s)
               for w, m, s in zip(weights, mus, sigmas))

Chapter 2: Particle Distribution

What if your posterior is so complex that no named distribution can describe it? Maybe it's multimodal, skewed, or lives on a weird manifold. The particle distribution (empirical distribution) sidesteps the problem entirely: represent the distribution as a collection of N weighted samples (particles).

Each particle x_i is a specific value, and w_i is its weight (with ∑w_i = 1). The δ is a Dirac delta — a spike at that point. More particles in a region means higher density there. This is the foundation of particle filters and Sequential Monte Carlo (SMC).

The critical operation is resampling: when particle weights become uneven (a few particles hog all the weight), you duplicate high-weight particles and discard low-weight ones. This keeps the approximation healthy.

Chapter 3: Spike-and-Slab Prior

You're fitting a regression model with 10,000 features, but you suspect only 50 actually matter. You want the model to discover which features are relevant and set the rest to exactly zero. This is variable selection, and the spike-and-slab prior is the Bayesian gold standard for it.

The spike is a point mass at zero — it says "this coefficient is exactly zero, the feature is irrelevant." The slab is a broad Gaussian — it says "this coefficient is nonzero, the feature matters." The mixing weight π controls the prior probability that any given feature is relevant.

Chapter 4: The Horseshoe Prior

Spike-and-slab is theoretically beautiful but computationally expensive — you need to explore 2^p possible inclusion patterns. The horseshoe prior achieves similar sparsity with a continuous distribution, making it far more practical for high-dimensional problems.

The trick is the global-local structure. The global scale τ controls overall shrinkage (small τ = most coefficients near zero). Each coefficient also gets a local scale λ_j drawn from a half-Cauchy. The heavy tails of the Cauchy let truly important coefficients escape the shrinkage.

Chapter 5: The Normal-Gamma Prior

Here's a common problem: you're observing data from a Gaussian, but you don't know the mean or the variance. You need a joint prior over both. The Normal-Gamma distribution is the conjugate prior for exactly this situation.

Here λ = 1/σ² is the precision (inverse variance). The prior says: first draw a precision λ from a Gamma, then draw the mean μ from a Gaussian whose variance depends on λ. This coupling is what makes the math work out.

After observing data, the updates are elegant: κ_n = κ₀ + n, μ_n is the precision-weighted average of prior and sample means, α_n = α₀ + n/2, and β_n incorporates the sample variance and the prior-data disagreement.

Chapter 6: The Logistic-Normal

You need a distribution over probability vectors — like topic proportions in a document (30% politics, 50% sports, 20% tech). The Dirichlet distribution is the standard choice, but it has a limitation: its components are negatively correlated by construction. If topic A increases, the others must decrease. What if topics tend to co-occur?

The Logistic-Normal distribution fixes this. Draw a vector from a multivariate Gaussian, then push it through the softmax function. The Gaussian's covariance matrix lets you model arbitrary correlations between components.

Chapter 7: The Truncated Gaussian

A standard Gaussian assigns nonzero probability to the entire real line. But what if your parameter can't be negative? A length, a concentration, a time duration — these are strictly positive. Naively clamping a Gaussian to [0, ∞) changes its mean and variance in hard-to-track ways.

The truncated Gaussian solves this cleanly: take a normal distribution N(μ, σ²) and restrict it to an interval [a, b], then renormalize so the density integrates to 1.

Here φ is the standard normal PDF and Φ is the CDF. The denominator is just the probability that an untruncated sample would fall in [a, b]. The shape stays Gaussian within the interval, but the effective mean shifts toward the interval center.

Chapter 8: The Folded Normal

You're measuring the magnitude of some quantity — the absolute deviation from a target, the strength of a signal, the distance a robot drifted from its path. You know the underlying error is Gaussian, but you only observe the absolute value. The result is a folded normal distribution.

The PDF is the sum of the original Gaussian and its mirror image reflected at zero:

When μ = 0, the folded normal simplifies to the half-normal distribution — a popular default prior for scale parameters in Bayesian hierarchical models (recommended by Andrew Gelman as a weakly informative prior for standard deviations).

Chapter 9: The Gumbel Distribution

Imagine measuring the maximum daily temperature every year and asking: "What's the distribution of the annual maximum?" This is an extreme value problem, and the Gumbel distribution is one of the three possible limiting distributions for maxima of independent samples.

The Gumbel has a distinctive asymmetric shape: a steep rise on the left and a long tail on the right (for the maximum case). But its most famous modern use has nothing to do with weather.

The math: if g_k ~ Gumbel(0, 1) independently, then argmax_k(log π_k + g_k) gives a sample from Categorical(π). Replace argmax with softmax at temperature τ for a differentiable relaxation.

Chapter 10: The Kumaraswamy Distribution

The Beta distribution is the conjugate prior for Bernoulli and binomial likelihoods, and it's ubiquitous in Bayesian statistics. But it has a dirty secret: its CDF has no closed form. You need the incomplete Beta function, which requires numerical approximation. For variational inference, where you need to compute KL divergences and sample using the reparameterization trick, this is a real problem.

Enter the Kumaraswamy distribution: a two-parameter distribution on [0, 1] that looks like a Beta but has closed-form PDF, CDF, and inverse CDF (quantile function).

python
import numpy as np

def kumaraswamy_sample(a, b, n=1000):
    """Sample via inverse CDF — no special functions!"""
    u = np.random.uniform(0, 1, n)
    return (1 - (1 - u)**(1/b))**(1/a)

Chapter 11: When to Use What

You've now met ten distributions that power modern Bayesian inference. Each solves a specific structural problem that standard distributions can't handle. Here's your decision guide.

Conjugacy Cheat Sheet

When your problem has a conjugate pair, use it — the posterior updates are free.

Every distribution in this lesson was born from a practical need. The Gaussian alone cannot model sparsity, boundedness, compositions, extremes, or arbitrary posteriors. Bayesian inference is as much about choosing the right distributional family as it is about applying Bayes' theorem.