Principal Components Analysis

Many neurons fire together, yet the activity of a population often moves along a few directions. PCA finds those directions and gives each one a number: how much activity it holds.

Learn how PCA turns a cloud of spike counts into an ellipse, why its axes are the best directions to look along, and how the SVD, eigenvectors and covariance fit together.

Drag the shared drive, then choose which direction to look along. Watch one direction hold far more of the activity than either neuron alone. Then we build the maths piece by piece: the size of a matrix, rank, eigenvectors, covariance, and the ellipse.

You need a spike count (how many spikes a neuron fires in one time window) and the idea of a matrix (a grid of numbers). We build the rest from zero.

Based on Lecture 3 of Princeton NEU 560, Statistical Modeling and Analysis of Neural Data (Spring 2024), taught by Jonathan Pillow. The slides and the lecture notes are on the course site. Every idea is re-taught here in our own words and devices.

Pick a direction to look along

0.30

Toy data, labelled illustrative: 60 time bins, spike counts of two made-up neurons shown as the change from each neuron’s average, seeded so every visit is the same. The shared drive is the correlation between the two neurons. The numbers are sums of squared projections (each point’s shadow on the line, squared and added), exactly as the lecture defines them.

Chapter 0

Why a Population Needs a Short List

See why a recording from many neurons can hide a much smaller story, and what a matrix of spike counts looks like

Picture a laboratory. A monkey reaches for a target, and a small array of electrodes in its motor cortex listens to 100 neurons at once. Every 20 milliseconds the equipment writes down how many spikes each neuron fired. After ten minutes you hold a long table of whole numbers.

That table is hard to look at. You cannot plot 100 neurons on one axis, and a plot with 100 wiggling lines tells you nothing. Yet you suspect the neurons are not acting alone. They share inputs, they excite one another, and they all serve one reaching arm.

So here is the question this lecture answers. Is the 100-number description necessary, or can a handful of numbers per time bin say almost the same thing? PCA is the tool that gives a rigorous answer.

One population, one table

A spike count is the number of spikes a neuron fires inside one short time window, called a time bin. Take all d neurons in the same bin and write their counts as a list. That list is one population vector: a single point in a space with one axis per neuron.

Record N bins and you have N such points. Stack them as rows and you get a matrix X (a grid of numbers) with N rows and d columns. In the lecture notes, each row is the response of d neurons to a single stimulus. The shape is N × d: bins down the side, neurons across the top.

Play with the shape below. The tall block is X. The small square is the matrix C = XTX, which the lecture will call the second moment matrix. The little T means transpose: flip the grid so rows become columns. We build C properly in Chapter 7.

See the shape of X and C

The tall block is X, one row per time bin and one column per neuron. The square is C = XᵀX. Drag both sliders and watch which block grows with which knob.

600
40

Illustrative sizes. Widths and the sides of C share one scale in neurons. The height of X uses a square-root scale of N, so that 3,000 bins still fit. The counts of numbers are exact: N × d for X and d × d for C.

Notice what the square does. Its size depends only on d, the number of neurons. Record ten times longer and X grows ten times taller, but C stays the same size. Most of what PCA needs lives in C.

The suspicion: the points do not fill the space

Each population vector can point anywhere in d dimensions. The lecture notes make a simple suspicion precise. The vectors may not “fill” the whole space. They may sit inside a smaller, flat piece of it, called a subspace.

The notes give the cleanest example. Suppose two neurons always fire the same number of spikes. Then every point has the form (x, x). All the points fall on the line where neuron A equals neuron B. That line is a one-dimensional subspace, and a single number per bin describes the data exactly.

Real neurons are never that perfect. But they are often partly shared. Then the points form a long, thin cloud around a line, not an exact line. The cloud is wide in one direction and narrow in the others. The hero at the top of the page shows exactly this for two neurons.

This asks for a theory of dimensionality reduction: describing d numbers with fewer numbers, and saying how much you lose. The notes put the question this way: how much of the variance (the spread of the data) does a low-dimensional projection capture? Yes, there is a rigorous answer, and the answer is PCA.

Where the lecture starts

Lecture 3 opens with a slide titled “SVD Leftovers”. The previous lecture built the singular value decomposition (SVD), a way to split any matrix into simple rank-one pieces. This lecture first tidies four loose ends, and then uses them to derive PCA. We follow the same order, and we re-teach the SVD pieces we need as we go.

  1. Low-rank approximation: a neuron-by-time matrix as a sum of simple pieces.
  2. The Frobenius norm: the size of a matrix, and the fraction of variance.
  3. What the SVD reveals: rank, column space, row space and null space.
  4. Eigenvectors: directions a matrix only stretches.
  5. Positive semi-definite matrices and the spectral theorem.
  6. Determinants: how a matrix changes volume.
  7. The best one direction: the first principal component, derived.
  8. k principal components and the fraction of variance.
  9. PCA as an ellipse, and why you must centre the data.
  10. Field guide: numbers, checklist, code, and where it leads.
Two neurons fire exactly the same spike count in every time bin. What does a plot of neuron A against neuron B look like, and what does that mean?

Chapter 1

A Neuron-by-Time Matrix as a Sum of Simple Pieces

Rebuild a recording from a few rank-one pieces, and see why keeping the first k pieces gives the best short version

You have recorded 8 neurons for 60 time bins. Plot them as eight rows of heat: warm where a neuron fired above its average, blue where it fired below. You see stripes. Some neurons rise and fall together, and others do the opposite.

The stripes repeat. Neuron 1 looks like a louder copy of neuron 2. Neuron 7 looks like an upside-down copy. That repetition is a hint that the table contains far less news than its 480 numbers suggest.

One piece: who, and when

Lecture 3 begins by recalling the picture from last time. The matrix A has one row per neuron and one column per time bin. The lecture draws it as a sum of simple matrices. Each simple matrix is built from two short lists.

The first list is a set of neural weights: one number per neuron, saying how strongly that neuron takes part. The second list is a timecourse: one number per time bin, saying when the pattern is on. Multiply every weight by every timecourse value and you get a full grid. This grid is a rank-one matrix, called an outer product.

Here is a tiny case with every step shown. Take 3 neurons and 4 time bins. The weights are (2, 1, 0) and the timecourse is (1, 0, −1, 0). Row 1 is 2 times the timecourse: (2, 0, −2, 0). Row 2 is 1 times it: (1, 0, −1, 0). Row 3 is 0 times it: all zeros.

(2, 1, 0)T × (1, 0, −1, 0) = rows (2, 0, −2, 0), (1, 0, −1, 0), (0, 0, 0, 0)

Twelve numbers came out of seven. Neuron 3 is silent in the pattern, neuron 1 is twice as active as neuron 2, and the pattern is on in bin 1 and off in bin 3. The whole grid is one idea, written once.

The sum, and the singular value decomposition

A real recording is not one pattern. It is several patterns added together, each with its own weights and timecourse. The SVD finds them for you. It writes A as a sum, and it orders the pieces from most important to least.

A
The matrix of d neurons by T time bins, one row per neuron.
s1
The first singular value: a number saying how big piece 1 is. The pieces are ordered, so s1 ≥ s2 ≥ and so on.
u1
The neural weights of piece 1: one number per neuron, scaled to length 1. The lecture draws these as “neural weights #1”.
v1T
The timecourse of piece 1: one number per time bin, scaled to length 1. The lecture draws it as “timecourse #1”.
sk uk vkT
Piece k. Stop the sum here and you have a rank-k approximation of A.

Written as matrices, A = U S VT. U has the weight lists as columns, S is a diagonal grid of the singular values, and VT has the timecourses as rows. For d neurons and T bins, with r pieces, the shapes are d×r, r×r and r×T.

Now the claim from the lecture slide. Keep the first k terms and stop. Among all matrices with only k pieces, this one is closest to A in squared error (the sum of the squared differences between matching entries). Nothing else with rank k does better. This is why the order matters: the SVD puts the most important pieces first.

Try it. The top block is the toy recording. Move the slider to choose how many pieces to keep. The middle block is the rebuild from those pieces. The bottom block is what the rebuild missed.

Rebuild the recording from k pieces

Top: the recording, 8 neurons by 60 bins. Middle: the rebuild from k pieces. Bottom: what is left over. Warm is above average and blue is below. Drag k from 0 to 8.

2

Toy data, illustrative: two hidden signals (a slow wave and a bump) mixed into 8 neurons with fixed weights, plus noise, with each neuron’s average removed. The SVD is computed live in your browser. The leftover is A minus the rebuild, measured as a sum of squares.

At k = 0 the rebuild is blank and everything is leftover. At k = 1 the slow wave appears. At k = 2 the bump joins, and the leftover looks like speckle. From k = 3 onward, each new piece mostly copies noise.

Here are the real numbers for the toy recording. Piece 1 alone holds 75.1% of the sum of squares. Piece 2 adds 12.2 points, which brings the total to 87.3%. That is why the rebuild at k = 2 already looks like the recording. The remaining six pieces share the last 12.7%, about 2 points each.

The failure mode

Symptom: keep one piece and the bump in the recording vanishes. The rebuild is clean but wrong. Measurement: the leftover block still shows a structured stripe, not speckle, and its sum of squares is large. Fix: keep more pieces until the leftover looks like noise. But how large a leftover is acceptable? The slide ends with exactly that question, “where does the fraction of variance come from?”, and Chapter 2 answers it.

The full proof that truncation is best is not in this lecture; it was last time’s result. For the maths of the SVD and PCA in depth, our PCA lesson goes step by step.

A rank-one matrix is built from two lists. What are they, for a neuron-by-time matrix?

Chapter 2

The Size of a Matrix and the Fraction of Variance

Measure how much activity a recording holds with one number, and learn why the singular values add up to it

Your colleague asks: “How much activity is in this recording?” You could say 480 numbers. But that is a count, not an amount. You want one number that grows when neurons fire harder and shrinks when they go quiet.

For one list of numbers, we already have such a number: its length. Square every entry, add them up, take the square root. That is the Euclidean norm. The lecture asks for the same thing for a whole grid.

Flatten the grid and measure it

Pull every entry of A out into one long list, and measure that list. The result is the Frobenius norm, written ||A||F. The slides call it “the Euclidean norm for matrices”. The square of it is just the sum of squared entries.

Take the small matrix with rows (1, 2) and (3, 4). The squares are 1, 4, 9 and 16. They add to 30, so ||A||F2 = 30. Hold on to this example. We will meet it three times.

Slide 5 shows a second way to get the same 30. Read the matrix by columns. Column 1 is (1, 3), and its dot product with itself is 1×1 + 3×3 = 10. Column 2 is (2, 4), and its dot product with itself is 4 + 16 = 20. The two dots add to 30 again.

Now collect every dot product between every pair of columns into a grid. That grid is ATA. Its diagonal holds the columns dotted with themselves: 10 and 20. The trace of a matrix is the sum of its diagonal, so Tr(ATA) = 10 + 20 = 30. The sum of squared entries is a trace.

The SVD shortcut

Slide 6 now spends the SVD, A = U S VT. Put it into ATA. The transpose of A is V S UT (S is diagonal, so it equals its own transpose). The product is V S UT U S VT.

Here the middle part simplifies. The columns of U have length 1 and are at right angles, so UTU is the identity grid and vanishes. What remains is V S2 VT. The slide labels this step “simplify”.

Next, the trace has a handy property: you may move the last factor to the front without changing it. The slide calls this the “circular shift”. So Tr(V S2 VT) = Tr(S2 VT V). The same trick turns VTV into the identity, leaving Tr(S2): the sum of the squared singular values.

||A||F
The Frobenius norm: the length of A when you flatten it into one long list.
Σij aij2
Square every entry and add them all up. The slide calls this the sum of squared elements of A.
Tr(ATA)
The sum of the diagonal of ATA, where each diagonal entry is a column dotted with itself.
Σi si2
The sum of the squared singular values of A. The slide calls this the sum of squared singular values.

Check it on our example. The matrix with rows (1, 2) and (3, 4) has singular values 5.465 and 0.366. Their squares are 29.866 and 0.134, and the sum is 30.000. The sum of squared entries was 30. The two routes agree.

What this buys you

Because the total activity is the sum of squared singular values, each singular value owns a share. Piece i owns si2. Keep the first K pieces and you keep a share given by the lecture’s fraction:

fraction of variance = (s12 + … + sK2) / ||A||F2

The slide calls the top the “sum of squared first K singular values” and the bottom the “sum of squares of all singular values”. It answers Chapter 1’s question. The same identity also says what the rebuild throws away: the leftover has a sum of squares equal to sK+12 + … + sr2.

In this lecture the word variance means sum of squares, with no mean removed yet. Chapter 9 explains why you usually subtract the mean first. The toy recording here already has each neuron’s average removed.

Share out the sum of squares

Each bar is si2, the share of the recording that piece i owns. Teal bars are kept, purple bars are thrown away. Choose how noisy the recording is, then drag K.

2

Toy data, illustrative: the same eight-neuron recording as Chapter 1 with noise of 0.3, 0.7 or 1.5 spikes. Both totals are computed live: one from the entries, one from the singular values. They agree to the digits shown.

With quiet data, two bars tower over the rest and K = 2 keeps almost everything. With noisy data the bars flatten, and no short list captures most of the recording. The shape of these bars is the first thing to look at in any real dataset.

The failure mode

Symptom: a report says the first piece holds “94%” of the recording, but the rebuild looks poor. Measurement: the author divided s1 by the sum of the singular values, not their squares. For our (1, 2), (3, 4) example that gives 5.465 / 5.831 = 93.7%, but the true share is 29.866 / 30 = 99.6%. Fix: square first. Squares match the sum of squared entries; plain values do not.

A recording has singular values 6, 3 and 2. What fraction of its sum of squares does the first piece hold?

Chapter 3

What the SVD Reveals: Rank and the Three Spaces

Count the independent patterns in a recording, and find the exact directions a matrix uses and ignores

Four neurons are recorded across six time bins. When you look at the numbers, neuron 4 looks suspicious. In every bin, its count equals neuron 1 plus neuron 2. It adds no news. If you deleted it, nobody could tell from the other three.

Real recordings have such redundancy in disguise. A neuron may be 1.7 times another plus a third. You cannot see this by eye in 100 rows. You need a test that counts the truly independent rows, and the SVD gives it.

Question 0: the rank

The rank of a matrix is the number of independent patterns it holds. A row is independent if no mix of the other rows can rebuild it. In our toy case the rank is 3, not 4, because neuron 4 is the sum of two others.

The lecture’s in-class answer is short: the rank is the number of non-zero singular values. Each non-zero singular value is one real piece of Chapter 1’s sum. A piece with s = 0 contributes nothing. Count the real pieces and you have the rank.

In practice, computers carry rounding error, so a singular value that should be 0 comes out as 0.000000000000001. You therefore count values larger than a small tolerance. Our device uses 10−9 times the largest value.

Questions 1 to 3: three spaces

A space here means all the vectors you can build by mixing a few chosen vectors. Mixing means multiplying each by a number and adding. Three spaces come out of A = U S VT, and slide 11 reads them off the three factors.

The column space is every population pattern that A can produce: all mixes of its columns. An orthonormal basis is a set of vectors of length 1 at right angles to each other that spans it. The answer is the first r columns of U, where r is the rank. In neural terms, this is the set of whole-population patterns that can ever show up in a time bin.

The row space is all mixes of the rows: every timecourse that appears in some neuron. Its orthonormal basis is the first r columns of V, which are the first r rows of VT. In neural terms, these are the distinct timecourses your neurons actually follow.

The null space is the set of vectors x with A x = 0. Here x has one entry per time bin. A x is the weighted sum of the columns of A, so a null vector is a way to weight the time bins so that all population patterns cancel exactly.

The answer is the remaining columns of V: vr+1 to vT. The slide gives the reason. Any mix of those has zero dot product with v1 to vr. A only reads the directions v1 to vr, so the other directions return zero.

The sizes follow at once. For a d by T matrix of rank r, the column space has r dimensions inside d, the row space has r inside T, and the null space has T − r.

Count the real patterns

The grid is 4 neurons by 6 time bins. Choose how much redundancy it has. The bars are its singular values. Then press Try a mix to test the null space.

Toy counts, illustrative. A singular value counts as zero below 10−9 times the largest. The null vectors come from the eigenvectors of AᵀA, and each Try a mix draws a fresh random mix of unit length.

With all neurons independent, the rank is 4, so four bars are non-zero and the null space has 6 − 4 = 2 dimensions. Make neuron 4 a copy of 1 plus 2 and one bar drops to zero. The rank falls to 3 and the null space grows to 3 dimensions. Press Try a mix: A times any mix of null vectors has length 0.000000, while a mix of the row-space vectors does not.

Real data never gives an exact zero. Rounding error and noise make every singular value a little above 0. So “count the non-zero values” becomes “count the values above a tolerance”. This count is the numerical rank. Choose the tolerance by looking at the bars: a clear cliff means a clear rank, and a smooth slope means you must decide.

The failure mode

Symptom: a decoder trained on 100 neurons gives wild weights, and a tiny change in the data swings them. Measurement: the singular values of the neuron matrix fall to nearly zero at some index, so the matrix has a much smaller rank than 100, or an almost-zero singular value. Fix: work in the span of the pieces with real size, which is the column space, or keep only the top r pieces. PCA does exactly that.

A 4 by 6 matrix of neuron counts has three non-zero singular values. What is the dimension of its null space, the set of x with A x = 0?

Chapter 4

Eigenvectors: Directions a Matrix Only Stretches

Find the special directions where a square matrix scales a vector without turning it, and ask when they match the singular vectors

Imagine two neurons that excite each other. Neuron A’s activity feeds into neuron B, and B feeds back into A. You start the pair at some state, a point in the plane of their two rates, and let the loop run one step. The state moves to a new point.

Most starting points swing round to a new direction as they move. A rare few do not turn at all. They only grow or shrink along the line they started on. Those special lines tell you how the loop behaves, and they are the subject of the second part of the lecture.

Slide 12 introduces four “bonus topics”, all about the SVD of square matrices: eigenvectors, positive semi-definite matrices, the spectral theorem and determinants. This chapter covers the first.

The definition

A square matrix has as many rows as columns, so it maps a list of d numbers to another list of d numbers. Call the matrix A and the list x. The product A x is a new list of the same length. Usually A x points somewhere else than x does.

An eigenvector of A is a non-zero vector x for which A x is a scaled copy of x. Slide 14 writes it as one line, with the scale factor labelled:

A
A square matrix, such as the connection strengths in a small loop of neurons.
x
The eigenvector: a direction that A does not turn. The slide labels it “eigenvector”.
λ
The eigenvalue: how much A stretches x. If λ is 2, the vector doubles. If λ is −1, it flips. The slide labels it “eigenvalue”.
x
The same vector again. A x is “a scaled version of x”, in the slide’s own words.

Take the matrix with rows (3, 1) and (1, 2). Try x = (1, 0). Then A x = (3, 1), which points up and to the right, so x is not an eigenvector. Now try a direction near 31.7 degrees, x = (0.851, 0.526). A x = (3.08, 1.90), and 3.08 / 0.851 = 3.62 and 1.90 / 0.526 = 3.62. Both ratios match, so A x = 3.62 x. The direction is an eigenvector, and 3.62 is its eigenvalue.

Hunt for them

The device draws x as a teal arrow of length 1, and A x as a warm arrow. Turn x with the slider. When the two arrows line up, you have found an eigenvector. A dashed warm line marks each true eigen-direction, and a purple tick marks the first singular vector of the same matrix.

Find the directions that do not turn

Teal: your vector x. Warm: A x. Drag the angle until the arrows line up. Then switch the matrix and compare the dashed eigen-lines with the purple singular-vector tick.

10°

The three matrices are toy examples: rows (3, 1), (1, 2) for Symmetric; rows (2, 1), (0, 1) for Not symmetric; and rows (0, −1), (1, 0) for a quarter turn. Eigenvalues and eigenvectors are computed live. Only angles from 0 to 180 degrees are shown, because x and −x lie on the same line.

With the symmetric matrix, two dashed lines appear, at 31.7 and 121.7 degrees. They are at right angles, and they sit exactly on the singular-vector ticks. With the second matrix, the eigen-lines are at 0 and 135 degrees. They are not at right angles, and they do not match the singular vectors. With the pure turn, no arrow ever lines up, because a quarter turn rotates every direction.

This answers the slide’s own question: when is an eigenvector equal to a singular vector? It happens only for special matrices. Chapter 5 names them.

What repeated steps do

Eigenvectors earn their name in loops. Feed the output of the pair back in as the next input, and apply A again and again. Along an eigenvector, each step multiplies the state by λ. After n steps the state is λn times the start.

If |λ| is above 1, the activity along that line grows with every step: runaway excitation. If |λ| is below 1, it shrinks to nothing. For the symmetric matrix above, the 31.7 degree line has λ = 3.62, so after three steps it has grown 3.62 × 3.62 × 3.62 = 47 times. The 121.7 degree line has λ = 1.38 and grows only 2.6 times.

Any other start is a mix of the two lines. The mix soon tilts toward the line with the bigger λ, because that part outgrows the other. So the eigenvectors with the largest eigenvalues describe where the activity ends up.

Why this matters for PCA

PCA needs the directions in which a cloud of spike counts is widest. Those turn out to be eigenvectors of the data’s covariance matrix. For such matrices, eigenvectors and singular vectors are the same thing, so the SVD can find them.

The failure mode

Symptom: you compute the eigenvectors of a connection matrix and expect them to be at right angles, but the angle between them is 45 degrees. Measurement: check whether the matrix equals its transpose. Our second matrix does not. Fix: only for symmetric matrices are eigenvectors at right angles and equal to singular vectors. For other matrices use the SVD for orthogonal directions, and use eigenvectors only to study repeated application.

For the symmetric matrix with rows (3, 1) and (1, 2), the vector x = (0.851, 0.526) gives A x = (3.08, 1.90). What does that tell you?

Chapter 5

Positive Semi-Definite Matrices and the Spectral Theorem

Learn why covariance matrices never have negative variance, and why for them the SVD is also an eigen-decomposition

Two neurons have a correlation of 0.8. You write their relationship as a small grid with 1 on the diagonal and 0.8 off it. That grid is a perfectly good description of how the pair moves together. Now someone claims the correlation is 1.5. The grid looks just as tidy. But no real pair of neurons can behave that way.

Something in the maths must say “1.5 is impossible”. That something is the idea of a positive semi-definite matrix. It is the property every covariance matrix has, and it is the second bonus topic of the lecture.

Positive semi-definite

Slide 15 gives two equivalent definitions. A positive semi-definite (PSD) matrix is one whose eigenvalues are all at least 0. The second definition: for any vector x, the number xT A x is at least 0.

The second form has a meaning. For a symmetric matrix A, the number xTA x (called a quadratic form) asks how big A is when you look along x. PSD means “never negative, whichever direction you look”. A variance is a size of spread, so it can never be negative. That is why covariance matrices are PSD.

The slide also shows how to make one. Take any matrix X with n columns and let A = XTX. Then xTA x = xTXTX x = (Xx)T(Xx), the squared length of the vector Xx. A squared length is at least 0, so A is PSD, always.

Now see why this is central for neural data. When X is the N by d matrix of spike counts, C = XTX is d by d, and xTC x is the sum of squared projections of every population vector onto the direction x. Chapter 7 builds PCA on exactly this number.

Drag the device below. The matrix has 1 on the diagonal and b off it, so it looks like a correlation matrix. The plot shows xTA x for every direction x around a circle: warm where it is positive, red where it dips below zero.

Look along every direction

The matrix is rows (1, b) and (b, 1). The petal plot shows xᵀAx for every direction x. Drag b from 0 past 1 and watch what happens along the 135 degree line.

0.80

A toy matrix, illustrative. The form is xᵀAx = 1 + b sin 2θ, where θ is the angle of x. The eigenvalues are 1 + b along 45 degrees and 1 − b along 135 degrees, computed live. With 1 on the diagonal, the matrix is a valid correlation matrix only when b is between −1 and 1.

Take b = 0.8. The eigenvalues are 1.80 and 0.20. Both are above 0, so the matrix is PSD, and the petal plot never goes below 0.20. The largest value, 1.80, sits along the 45 degree line, and the smallest sits along the 135 degree line. Those two lines are the eigenvectors, at right angles. Push b to 1.5 and the eigenvalues become 2.50 and −0.50. Now a red petal appears along 135 degrees: a negative “variance”. The matrix is not PSD, and no real neurons could produce it.

The spectral theorem

Slides 16 and 17 give the payoff. If a matrix A is symmetric (it equals its own transpose) and positive semi-definite, then its SVD is also an eigen-decomposition, a split into stretch directions and stretch amounts:

A
A symmetric, positive semi-definite matrix, such as a covariance matrix.
U
A matrix whose columns are the eigenvectors, each at a right angle (orthogonal) to the others. The slide labels it “matrix of (orthogonal) eigenvectors”.
S
A diagonal matrix with the eigenvalues along its diagonal. The singular values are the eigenvalues.
UT
The transpose of U. In the general SVD this slot holds VT. Here V equals U, so the left and right singular vectors are the same.

So for these matrices three pairs of words mean the same thing: singular vectors and eigenvectors, singular values and eigenvalues, left and right singular vectors. This also answers Chapter 4’s question. An eigenvector equals a singular vector when the matrix is symmetric and PSD.

Slide 17 sets an exercise: prove that A ui = si ui. Here is the proof in four steps. First, A ui = U S UT ui. Second, UTui is a list of zeros with a 1 in slot i, because the columns of U are orthonormal. Third, S times that list gives si in slot i and zeros elsewhere. Fourth, U times that list picks out si times column i, which is si ui.

Slide 18 connects it all to a general matrix A. Compute ATA = (V S UT)(U S VT) = V S2 VT. The slide notes that V holds orthogonal eigenvectors and the s2 are the eigenvalues, and it writes beside it: covariance matrices! For our (1, 2), (3, 4) example, ATA has rows (10, 14) and (14, 20). Its eigenvalues are 29.866 and 0.134, the squares of Chapter 2’s singular values 5.465 and 0.366.

The failure mode

Symptom: you build a covariance matrix by hand from pairwise correlations that look reasonable, and a later step takes the square root of a variance and fails. Measurement: the smallest eigenvalue is below zero, such as −0.50. Fix: estimate the matrix from data as XTX, which is PSD by construction, or set negative eigenvalues to zero.

The matrix with rows (1, 1.5) and (1.5, 1) has eigenvalues 2.5 and −0.5. What does the negative eigenvalue tell you?

Chapter 6

Determinants: How a Matrix Changes Volume

Watch a matrix stretch a square into a parallelogram, and see why the area change is the product of the singular values

You record a pair of neurons many times, and plot each trial as a dot. The dots make a cloud. Now you ask: how much room does this cloud take up? A tight cloud means the pair is very predictable. A wide cloud means it is not. You want one number for the “size” of the cloud.

The last bonus topic gives that number. It starts from a simple question about squares, and ends with the singular values you already know.

The determinant is a volume factor

Take the unit square, with corners (0, 0), (1, 0), (0, 1) and (1, 1). Its area is 1. Multiply every point by a matrix A. The square becomes a parallelogram. The determinant, written det |A|, is the factor by which the area changes. Slides 19 and 20 call it the “change in volume” of a hypercube. In higher dimensions the square becomes a cube and then a hypercube, and the area becomes volume.

The slide’s example is the matrix with rows (3, 0) and (0, 2). It stretches the first axis by 3 and the second by 2. The unit square becomes a 3 by 2 rectangle with area 6. So det |A| = 3 × 2 = 6.

If the matrix also shears, the rectangle leans into a slanted parallelogram. Its base and height stay the same, so the area stays the same. The determinant does not notice shear. It only notices how much room the shape takes.

What if the matrix squashes the square flat? Then the area is 0, and so is the determinant. That happens when the rows or columns are redundant, which is exactly when the rank drops (Chapter 3).

Using the SVD to compute it

Slide 21 shows how the singular values give the determinant. It applies to PSD matrices, where singular values equal eigenvalues. Write A = U S UT. Then det |A| = det |U S UT|.

Two facts finish the job. First, det |XY| = det |YX|, so you may move UT to the front: det |S UT U|. Second, UTU is the identity, with determinant 1. That leaves det |S|. The determinant of a diagonal matrix is the product of its diagonal entries.

det |A|
The factor by which A multiplies volume.
S
The diagonal matrix of singular values. The factors U and UT only rotate, and a rotation does not change volume.
∏ si
The product of the singular values. The slide labels it “product of singular values”.

The idea is plain in pictures. Any matrix is a turn, then a stretch along perpendicular axes, then another turn. Turns keep area. Only the stretches change it, and they multiply. The singular values are the stretch factors.

Stretch the square

The matrix has rows (a, c) and (0, 2). The blue square is the unit square, and the warm shape is what A turns it into. Drag a and c. The bars show the two singular values. Drag a to 0 to flatten the shape.

3.0
0.0

A toy matrix, illustrative. The start matches the slide: a = 3, c = 0 gives rows (3, 0), (0, 2). The determinant is a × 2. The singular values come from the eigenvalues of AᵀA, computed live. The general rule is that the size of the determinant equals the product of the singular values, for every square matrix; the slide states it for PSD matrices.

Leave c at 0 and the singular values are a and 2, with a product of 2a. Now add shear. The parallelogram leans, but its area does not change, so det stays 2a. The singular values do change, because the shape’s longest and shortest stretches change, but their product stays fixed. Drag a to 0 and the shape collapses to a line segment: det = 0, and one singular value is 0.

One more detail. The determinant can be negative. A negative value means the matrix also flips the shape, like a mirror. The size of the area change is still the absolute value, and that is what the product of the singular values gives. For the PSD matrices of PCA, the determinant is never negative.

The neural reading

Apply this to a covariance matrix C of d neurons. Its eigenvalues are the spreads along the PC directions. Their product, det C, is the squared volume of the ellipse that fits the cloud, up to a fixed factor. It is a single number for “how much room the population uses”. For the PSD matrix with eigenvalues 1.80 and 0.20 from Chapter 5, det = 1.80 × 0.20 = 0.36.

If two neurons are exact copies, one eigenvalue is 0 and det C = 0. The cloud is a flat line, and the population has lost a dimension.

The failure mode

Symptom: for 200 recorded neurons, a program reports det C = 0.0 exactly, although no neuron is redundant. Measurement: all 200 eigenvalues are about 0.01, so the product is 10−400, below the smallest 64-bit number (about 10−308), and it underflows to 0. Fix: sum the logarithms of the eigenvalues. Here log10 det C = 200 × (−2) = −400, a perfectly ordinary number.

A PSD matrix has singular values 3, 2 and 0.5. What is its determinant, and what does it mean?

Chapter 7

The Best One Direction: The First Principal Component

Derive the single direction that captures the most activity, and see it is the top eigenvector of C = XᵀX

Two neurons are recorded for many time bins, and each bin is a dot. You may keep only one number per bin. That number is the dot’s position along a line you choose, as if you shone a light and kept only the shadow. Which line keeps the most of what the dots were doing?

Slide 22 says “derivation on board”, and the lecture notes carry that derivation. We follow the notes step by step, with a small example, because every later step rests on this one.

Setting it up

The data are N vectors x1 to xN, one per time bin, each with d entries. They are the rows of X. A direction is a unit vector v: a vector of length 1, so vTv = 1. The shadow of xi on v is the dot product xi · v, a single number.

We want the v that makes the shadows as large as possible overall. Big positive and big negative shadows both count, so we square them and add. The notes call it the sum of squared linear projections, and they rewrite it step by step:

Σi (xi · v)2
Take each bin’s population vector, drop its shadow on v, square it, and add over all N bins.
||X v||2
X v is the list of all N shadows. Its squared length is the same sum.
vT XTX v
Squared length of a list is that list dotted with itself, and (X v)T = vTXT.
vT C v
With C = XTX, a d by d grid that does not grow with N. This is the quadratic form from Chapter 5.

A worked case shows every step. Take three bins and two neurons: x1 = (2, 1), x2 = (−1, −1), x3 = (−1, 0). Then C has top-left 4 + 1 + 1 = 6, off-diagonal 2 + 1 + 0 = 3, and bottom-right 1 + 1 + 0 = 2. So C has rows (6, 3) and (3, 2).

Look along neuron A, v = (1, 0): the shadows are 2, −1, −1, and the sum of squares is 6. Along neuron B, v = (0, 1): 1, −1, 0, giving 2. Now look along the direction at 28.2 degrees, v = (0.882, 0.472). The shadows are 2.235, −1.354 and −0.882, and their squares add to 7.606. A tilted view beats both neuron axes.

Turn the line, watch the shadows

Top: a cloud of two neurons, a teal line you can turn, and each dot’s shadow on it. Bottom: the sum of squared shadows at every angle. Drag the angle to the peak, or press Snap.

0°

Toy data, illustrative: 60 time bins of two made-up neurons with shared drive 0.75, with the means removed. The curve is vᵀCv = a cos²θ + 2b cosθ sinθ + c sin²θ, where C has entries a, b, c. Its peak is the top eigenvalue of C, and its low point is the other.

The curve is a smooth wave. Its peak is where the line points along the long axis of the cloud, and its low point is at right angles to it. The peak value is not just a high number. It equals an eigenvalue of C. The next section shows why.

Why the best direction is an eigenvector

Write C = U S UT, with orthonormal eigenvectors u1 to ud and eigenvalues s1 ≥ s2 ≥ … The lecture calls these sj. They are the squares of the singular values of X. First try v = uj only. Then UTuj is a list of zeros with a 1 in slot j. So ujT C uj = (that list)T S (that list) = sj.

Looking along eigenvector uj gives exactly sj as the sum of squared shadows. We want the largest, so choose the eigenvector with the largest eigenvalue, v = u1. This is the first principal component, or “first PC”. In our example, s1 = 7.606.

But we only tested eigenvectors. Could some other direction do better? The notes close the gap with the method of Lagrange multipliers. The idea: to maximise a function while a constraint holds, subtract a multiplier λ times the constraint. Write L = vTC v − λ(vTv − 1). At the best v, the slope of L must vanish.

The slope with respect to v is 2C v − 2λv. Setting it to zero gives C v = λv. That is the eigenvector equation from Chapter 4. So the best v must be an eigenvector, and among eigenvectors the best is the one with the largest eigenvalue. (The notes write the multiplier with the opposite sign, which only renames λ.)

The failure mode

Symptom: the first PC of a recording points almost entirely at one neuron. Measurement: that neuron has counts ten times larger than the rest, so its variance is a hundred times larger, and Cjj dominates the sum of squares. Fix: PCA measures spikes squared, so units matter. If you want every neuron to count equally, rescale each neuron by its own spread before you compute C.

For C with eigenvalues 7.6 and 0.4, what is the sum of squared projections when you look along the eigenvector with eigenvalue 7.6?

Chapter 8

k Principal Components and the Fraction of Variance

Keep the top k directions of a whole population, rebuild the activity from them, and read off how much you kept

Eight neurons, sixty time bins. You have found the best single direction, and it holds most of the activity, but not all. A second line at a right angle to the first may catch more. So may a third. How many lines do you need before the picture is good enough?

Slides 23 to 25 give the whole recipe on one page, and the notes explain each part. We build it in order, with the shapes of every object named, so you could write the code yourself.

The recipe, with shapes

Start with the data matrix X, N×d: one row per time bin and one column per neuron. Form the second moment matrix C = XTX, d×d. Take its SVD, C = U S UT. Because C is symmetric and PSD (Chapter 5), this is also its eigen-decomposition.

The first k columns of U, u1 to uk, are the first k principal components (PCs). Each is a direction in neuron space with d entries. They are orthonormal: each has length 1, and each pair is at right angles. Stack them as a matrix B, d×k.

Now the data flow. The product X B, N×k, has one row per time bin and k columns. Each row is that bin’s coordinates along the k PCs. This is the low-dimensional version of the recording: k numbers per bin in place of d. The product X B BT returns to N×d. It is the rebuild of X from k PCs.

The notes show the singular values have a direct meaning: sj = Σi (uj · xi)2. Eigenvalue sj is the sum of squared shadows of all bins on PC j, just as Chapter 7 showed for j = 1. The lecture’s sj here are the eigenvalues of C, the squares of the singular values of X.

fraction
The fraction of the total variance accounted for by the first k PCs. The slide calls the bottom line “fraction of sum of squares”.
s1 + … + sk
Sum of squares of the data inside the k-dimensional subspace. Each sj is the shadow energy on PC j.
s1 + … + sd
The sum of squares of all the data, which is ||X||F2. The slide boxes it as “sum of squares of all data”.

This is the same fraction as in Chapter 2, because the trace of C equals the sum of squared entries of X. It is also the same computation as Chapter 1. The PCs are the neural weight lists uj of the neuron-by-time matrix, and the coordinates in X B are the timecourses scaled by their singular values.

Count the savings in the toy case. The recording has 60 bins and 8 neurons, so X holds 480 numbers. With k = 2, the scores hold 60 × 2 = 120 numbers and B holds 8 × 2 = 16. Together that is 136 numbers, and they rebuild 87.3% of the sum of squares.

Two ways to say “best”

The notes give two equivalent goals. The first: choose B with orthonormal columns that makes ||XB||F2 as large as possible. The second: choose B to make ||X − XBBT||F2 as small as possible. The first keeps the most activity. The second leaves the least error in the rebuild.

They agree because the rebuild and the leftover are at right angles to each other, so their squared sizes add up to the total, which does not change. Large kept share means small leftover. Pythagoras for matrices.

Use the device. The bars are the sj of the toy population, with a line for the running total. The lower plot shows one neuron’s real activity in blue and its rebuild from k PCs in warm.

Keep k directions

Top: the eigenvalues sj of C, teal if kept. Bottom: one neuron over 60 bins, the data (blue) and the rebuild X B Bᵀ (warm). Pick a neuron and drag k.

2

Toy data, illustrative: the same 8 neurons by 60 bins as Chapters 1 and 2, as a 60 by 8 matrix X with each neuron’s average removed. The rebuild is X B Bᵀ, with B the first k eigenvectors of C = XᵀX. The error is the root mean square difference in spikes.

At k = 1 the rebuild follows the slow wave but misses the bump. At k = 2 it gets the bump too, and 87.3% of the sum of squares is kept. Adding a third PC raises the share only to 90.6%, and each later one adds only 1 to 3 points. The bars show the elbow after two directions. That is the hint that this recording is two-dimensional plus noise.

The failure mode

Symptom: you analyse the same neurons on two days, and PC 1 looks like a mirror image: every weight has flipped sign. Measurement: the dot product of the two PC 1 vectors is −0.99. Fix: the sign of an eigenvector is arbitrary, because both u and −u satisfy C u = s u. Fix the sign by a rule, for example make the largest weight positive, before you compare days.

C has eigenvalues 6, 3, 1 and 0. What fraction of the variance do the first two PCs account for?

Chapter 9

PCA as an Ellipse, and Why You Centre the Data

See the principal components as the axes of an ellipse around the cloud, then subtract the mean so the ellipse sits on the data

Draw the spike counts of two neurons as a cloud of dots. The cloud is not round. It stretches along a slanted line, like a grain of rice. Draw an oval round it, and the oval tells you the cloud’s whole shape with two lengths and one angle.

The last slides of the lecture say exactly this: PCA is equivalent to fitting an ellipse to your data. They build the picture in five steps, and this chapter walks through them.

Reading the ellipse

Slide 26 shows the cloud with its ellipse, on axes labelled dimension 1 and dimension 2. Slide 27 adds an arrow, u1, along the long axis. It is the first PC. Slide 28 marks the length of that axis with the singular value s1. Slides 29 and 30 add the short axis, u2, at a right angle, with its own length s2.

Slide 30 states the rule in two lines. The PCs are the major axes of the ellipse (or ellipsoid, the same oval shape in three or more dimensions). The singular values specify the lengths of the axes.

We can make the lengths exact. The sum of squared shadows on PC j is sj. Divide by the number of bins N and take the square root, and you get the typical size of a shadow, the root mean square (RMS) spread along that axis. So the semi-axis, the distance from the centre to the tip, is proportional to √(sj / N). Draw it at 1 RMS or 2 RMS, as you like.

Slide 31 collects the summary. PCA is a linear subspace capturing the maximal projected sum of squares. It is equivalent to fitting an ellipse. The singular vectors are the PCs, the major axes. The singular values are the amount of variance captured by each PC.

Where does the ellipse sit?

So far every sum of squares was measured from the origin. That means the ellipse is centred on the origin, at zero spikes for every neuron. But real spike counts have a typical level. A neuron that fires about 9 spikes a bin sits far from zero, and so does the cloud.

Then the ellipse centred at the origin has to stretch out to reach the cloud. Its long axis points at the cloud’s centre, and PC 1 mostly tells you where the average is, not how the cloud varies.

The notes fix this in section 3.3. We want an ellipse centred on the data, and PCs that describe the spread relative to the mean. So we centre the data first. The mean vector is the average over bins:

x̄
The mean population vector: the average spike count of each neuron over all N bins. It has d entries.
zi
Bin i with the mean removed. The matrix Z has these as rows, and ZTZ is the covariance matrix, up to a factor of N.

Take the SVD of ZTZ instead of XTX. The notes say plainly: this is the standard definition of PCA, and it is uncommon to do PCA on uncentred data. Our toy recording in Chapters 1, 2 and 8 had each neuron’s mean removed for this reason.

Try it. The cloud below has two neurons with mean counts of about 9 and 6 spikes per bin. Switch between raw and centred. The warm ellipse is 1 RMS wide along each PC, and the arrows show the two PCs.

Centre the cloud

Blue dots are 60 time bins of two neurons, in spikes. Choose raw or centred, and see where the ellipse sits and where PC 1 points.

Toy data, illustrative: the 60 bins of two made-up neurons that move in opposite directions (correlation −0.7), shifted to mean counts of 9 and 6 spikes. The ellipse is one RMS wide along each PC, with semi-axes √(sj/N). Raw uses XᵀX about the origin. Centred uses ZᵀZ about the mean.

With raw counts the ellipse starts at the origin and runs through the cloud. PC 1 points at about 32 degrees, almost the direction of the mean (34 degrees), and holds 85.8% of the sum of squares. Most of that share is the distance of the cloud from zero. It says nothing about how the two neurons move together. Switch to centred and the ellipse wraps the dots. PC 1 turns to 135 degrees, the line along which the neurons trade off, and holds 92.2% of the centred sum of squares.

The covariance in code

The notes mention that np.cov does the centring for you. One caution from us: NumPy treats rows as variables by default, and it divides by N − 1 unless you ask otherwise. For an N by d matrix of bins by neurons, call np.cov(X, rowvar=False). The notes’ division by N changes every eigenvalue by the same factor, so the PCs and the fractions do not change.

The failure mode

Symptom: PCA on raw spike counts gives a PC 1 with all-positive weights that track each neuron’s firing rate, and it holds a large share of the activity. Measurement: the angle between PC 1 and the vector of mean counts is only a few degrees (2 degrees in our toy). Fix: subtract the mean of each neuron before you form ZTZ, then check that the fractions have dropped to a sensible level.

You run PCA on raw spike counts, and PC 1 points almost exactly along the vector of mean firing rates. What went wrong, and what do you do?

Chapter 10

Field Guide: PCA on One Page

Keep the numbers, the checklist and the code, and see where to go next

Here is the whole lecture in one paragraph. A recording of d neurons over N bins is a matrix X. The size of X is its sum of squares, which equals the sum of the squared singular values. PCA centres X, forms C = ZTZ, and finds its eigenvectors. They are the axes of an ellipse fitted to the cloud. Each eigenvalue is the activity on its axis, and the running total tells you how many axes you need.

The numbers and terms

QuantityWhat it meansToy valueChapter
||A||F2Sum of squared entries, equal to the sum of squared singular values1,234.0 for the 8 by 60 toy recording2
RankNumber of non-zero singular values: independent patterns3 when neuron 4 = neuron 1 + neuron 23
Null spaceVectors with A x = 0, spanned by vr+1 to vT; dimension T − r3 dimensions for rank 3 and 6 bins3
A x = λ xEigenvector x is only scaled, by the eigenvalue λλ = 3.62 at 31.7 degrees4
PSDAll eigenvalues at least 0, or xTA x ≥ 0 for every x; true for every XTXEigenvalues 1.80 and 0.20 at b = 0.85
Spectral theoremFor symmetric PSD A, A = U S UT: singular vectors are eigenvectorsATA eigenvalues 29.866 and 0.1345
det |A|Volume factor; the product of the singular values6 for rows (3, 0), (0, 2)6
vTC vSum of squared projections on unit v; its maximum is s1, at v = u17.606 for the three-bin example7
Fraction of variance(s1 + … + sk) / (s1 + … + sd)87.3% for k = 2 of 88
Ellipse semi-axisProportional to √(sj / N), along PC jPC 1 turns from 32 to 135 degrees after centring9

The checklist

  1. Arrange the data as X, N bins by d neurons. Check the shape.
  2. Subtract each neuron’s mean to get Z. This is the standard form of PCA.
  3. If neurons have very different counts, decide whether to rescale them.
  4. Take the SVD of Z, or the eigen-decomposition of ZTZ. The eigenvalues are the squares of the singular values.
  5. Plot the eigenvalues and the running fraction. Look for the elbow.
  6. Choose k. Look at the leftover: it should look like noise.
  7. Fix the sign of each PC by a rule before you compare sessions.

The core as code

python
import numpy as np

def pca(X, k):
    # X: (N, d) spike counts, one row per time bin, one column per neuron
    mu = X.mean(axis=0)                       # (d,)  mean count per neuron
    Z = X - mu                                      # (N, d) centred data
    U, s, Vt = np.linalg.svd(Z, full_matrices=False)
    # rows of Vt are the PCs; eigenvalues of Z.T @ Z equal s**2
    B = Vt[:k].T                                    # (d, k) first k PCs
    scores = Z @ B                                  # (N, k) low-dimensional version
    rebuild = scores @ B.T + mu                     # (N, d) rebuild from k PCs
    frac = (s[:k] ** 2).sum() / (s ** 2).sum()  # fraction of variance kept
    return scores, B, rebuild, frac

# the same eigenvalues by the covariance route (notes: np.cov)
C = np.cov(X, rowvar=False)                    # (d, d), divides by N - 1
evals = np.linalg.eigvalsh(C)[::-1]                 # large to small; proportional to s**2

Where it leads

This lecture gives the tool; the lessons around it use it. Our PCA lesson goes deeper on the maths. The neural manifolds lesson in the brain-computer interface series uses PCA to describe population activity and its dynamics. If the low-dimensional state also moves smoothly in time, the Kalman filter lesson shows how to track it.

PCA treats spike counts as plain numbers with equal noise. Real counts are not like that: their spread grows with their mean. The generalised linear models lesson shows models that respect counts.

The takeaway. PCA finds the ellipse around your centred cloud. Its axes are the best directions to look along, and each axis comes with a number: how much of the activity it holds.

Credit: this lesson follows Lecture 3 of Princeton NEU 560, Statistical Modeling and Analysis of Neural Data (Spring 2024), by Jonathan Pillow. The course slides and lecture notes are on the course site. All explanations, examples and devices here are our own, and the toy data are illustrative.

Now explain PCA to someone else on the Teach whiteboard, or rehearse it in Present. Then go back to the hero and pick PC 1 once more. You now know why it wins.

NEU 560 Lecture 3: Principal Components Analysis
Course site · Back to Gleams