SVD and Its Applications

Sixteen neurons over 48 moments make a table of 768 numbers. Any such table is a rotation, a stretch and another rotation, and its few strongest stretches rebuild the whole recording.

Learn what the singular value decomposition is, why it gives inverses, ranks and best approximations for free, and how it finds the few patterns hidden in a population of neurons.

Press the buttons to rebuild the same recording from 1, 2, 3 or all 16 pieces. Watch how little of the recording each extra piece adds. Then we build, piece by piece, the vectors, the three steps, the inverse, the rank and the approximation underneath.

You need to know what a matrix is (a grid of numbers in rows and columns) and what multiplying a matrix by a vector does. We build the rest from zero.

Based on Lecture 2 of Princeton NEU 560: Statistical Modeling and Analysis of Neural Data (Spring 2024), taught by Jonathan Pillow: the slides (PDF) and the lecture notes on SVD. Every idea is re-taught here in our own words and devices. The course is the spine of our brain-computer interface series.

Rebuild a recording

A toy recording, not real data: 16 neurons by 48 time bins, built from three hidden patterns plus noise, then each neuron’s average removed (warm: above its average, blue: below). The pieces come from a real SVD computed in your browser.

Chapter 0

A Recording Is a Table of Numbers

See how a night of spikes becomes a matrix, and why one tool for taking matrices apart earns a whole lecture

Picture an electrode array in a monkey’s motor cortex. For one second it listens to eight neurons. Each neuron fires a few dozen spikes, at times that look random. You have 8 rows of event times and no obvious pattern.

Now you want to ask a question. Do the neurons move together? Is there one slow pattern that many of them share? Spike times do not answer that. You need numbers laid out so that you can compare neuron with neuron and moment with moment.

The standard move is to cut time into short slices and count the spikes in each. A slice is a time bin. The count in a bin, divided by the bin width, is a firing rate, in spikes per second. Put one neuron in each row and one time bin in each column. The result is a matrix: a grid of numbers with rows and columns. Call this one A.

Drag the bin width below. Notice that the matrix changes shape, but the same spikes feed it. Narrow bins make a wide, jagged table. Wide bins make a short, smooth one.

Bin the spikes into a matrix

Top: a raster, one row per neuron, one tick per spike. Bottom: the matrix of spike counts per bin. Drag the bin width and watch the matrix change shape.

50 ms

Illustrative spikes: eight toy neurons over 1 second, drawn from a seeded random stream. Neurons 1 to 4 share a rising rate, neurons 5 to 8 share a falling one. Not real data.

With 50 ms bins, one second of eight neurons is an 8 by 20 table: 160 numbers. A real recording is far bigger. A hundred neurons for ten minutes in 10 ms bins is a 100 by 60,000 table: six million numbers. No one reads six million numbers.

The one tool this lecture teaches

The lecture is about a single idea: the singular value decomposition, or SVD. It writes any matrix as three simpler matrices multiplied together. Each of the three does one easy job: turn, stretch, turn again.

Here is why that earns a lecture. Once a matrix is split that way, many hard questions become one-line answers. Can I undo this matrix? Look at the stretches. How many independent directions does it use? Count the stretches that are not zero. What is the best simple copy of it? Keep the biggest stretches. The slides open this part with tweets from statistician Daniela Witten, who tags the SVD the best matrix decomposition, and a thread arguing that every data scientist needs it. The lecture backs the joke with proofs.

The lecture builds the idea in a set order. It warms up with vectors and subspaces. It introduces the SVD on the whiteboard, and gives practice problems. It recaps the picture, then covers inverses, pseudo-inverses, rank, non-square matrices, the condition number and low-rank approximation. It ends on neural data. This lesson follows that order.

Housekeeping

This is Lecture 2 of Princeton’s NEU 560, Statistical Modeling and Analysis of Neural Data, taught by Jonathan Pillow in Spring 2024. The lecture has a slide deck and written notes. We re-teach every idea in our own words and our own devices. For the original slides and notes, see the slides PDF and the course site.

The road

  1. Vectors and projections: length, dot product, and the shadow of one vector on another.
  2. Subspaces, rank and null space: what a flat sheet through the origin is, and where neurons live on one.
  3. The SVD as three steps: rotate, stretch, rotate.
  4. Working with the three factors: the inverse, AAT and ATA.
  5. Inverse and pseudo-inverse: undoing a matrix, and what to do when you cannot.
  6. Rank and non-square matrices: counting directions, and the shapes of tall and wide tables.
  7. The condition number: when a matrix can be undone in theory but not in practice.
  8. Sums of outer products and low-rank approximation: the best simple copy of a matrix.
  9. SVD on neural data: timecourses and neural weights.
  10. The field guide: the numbers, the code and the way on.
You bin a 1-second recording of 8 neurons into 50 ms bins. What is the shape of the matrix A, and what happens if you halve the bin width?

Chapter 1

Length, Dot Product and Projection

Measure a vector, compare two vectors, and cast the shadow of one onto another

Two neurons are recorded for one moment. The first fires at 2 spikes per second and the second at 2 spikes per second. A moment later they fire at 5 and 0. Each moment is a pair of numbers, and a pair of numbers is a point on a sheet of graph paper. The first axis is neuron 1’s rate. The second axis is neuron 2’s rate.

An arrow from the origin to that point is a vector. The lecture opens with a warm-up on three of them: v1 = [2, 2], v2 = [5, 0] and v3 = [3, 4]. Think of each as the state of a two-neuron population. The warm-up asks three things about them. We take them in order.

How long is a vector?

The norm of a vector is its length. Walk 3 steps right and 4 steps up. The straight-line distance back to the start is 5, because 3² + 4² = 25 and the square root of 25 is 5. That is Pythagoras, and it is the whole definition: square every entry, add them up, take the square root.

In neural terms, the norm of a population vector says how strongly the whole population is firing at that moment, in one number. The three warm-up vectors have lengths √8 ≈ 2.83, 5 and 5. Notice that v2 and v3 are equally long but point in different directions.

‖v‖
The norm, or length, of the vector v.
v12
The first entry squared, for example the first neuron’s rate times itself.
v22
The second entry squared. With more neurons, you add one square per neuron.

Do two vectors point the same way?

The dot product multiplies matching entries and adds the results. For v1 = [2, 2] and v2 = [5, 0] it is 2·5 + 2·0 = 10. For v1 and v3 = [3, 4] it is 2·3 + 2·4 = 14.

What does the number mean? A large positive dot product says the two vectors lean the same way. Zero says they are at right angles: orthogonal. A negative one says they lean opposite ways. The exact link is: dot product = length of a × length of b × cosine of the angle between them.

The shadow of one vector on another

Now the third question. Shine a light straight down onto the line that v2 spans. The shadow of v1 on that line is its linear projection onto the one-dimensional subspace (a flat line through the origin) spanned by v2. The recipe: take the dot product, divide by the squared length of the line’s vector, and scale that vector.

Work it. v1·v2 = 10. The squared length of v2 is 25. The scale is 10/25 = 0.4. Then 0.4 × [5, 0] = [2, 0]. The shadow keeps the part of v1 that lies along the first neuron’s axis and drops the rest. This matches the lecture’s answer.

Cast a shadow

The fixed vector w = [5, 0] is the pink arrow. Set the length and the angle of your vector v (teal). The blue dashed arrow is the projection of v onto w.

45°
2.83

The defaults give the lecture’s v1 = [2, 2], with length 2.83 and projection [2, 0]. Values are exact for the vectors drawn.

Slide through 90 degrees. The projection shrinks to nothing at 90, because a vector at right angles to w casts no shadow. Past 90 it flips to point the other way, and the dot product turns negative.

Why this matters for the SVD

The SVD is built from these three tools. Its vectors have length 1, so their norms are easy. It uses dot products to say which directions are at right angles. And every time it “reads off a component”, it is projecting. When a later chapter says “the part of x along v1”, this is the dot product it means.

The vector [3, 4] is projected onto the line spanned by [5, 0]. What is the projection?

Chapter 2

Subspaces, Rank and the Null Space

Learn which sets of vectors count as flat sheets through the origin, and which matrices squash a direction to nothing

Three neurons fire together in a motor area before a reach. Their rates, taken as three numbers, are a point in a three-dimensional space. Over many trials the points do not fill that space. They cluster along a sheet inside it. A paper on premotor cortex drew exactly that picture, and the lecture shows it: three axes, one for each neuron’s firing rate, and shaded patches for the reach to the left and the reach to the right.

The lecture asks about that figure: are those shaded regions subspaces? Why or why not? To answer, we first need the definition. The rest of the warm-up fits around it.

What a subspace is

A vector space is a collection of vectors you can add together and scale, and still stay inside. A subspace is a smaller collection inside a bigger space that obeys the same two rules. Rule one: add any two members and the sum is a member. Rule two: multiply any member by any number and the result is a member.

One thing follows at once. Multiply any member by 0 and you get the zero vector. By rule two the zero vector must be in the set. So a vector space always contains the zero vector. That is the lecture’s answer to warm-up question 3: scalar multiplication by 0 has to stay inside the space.

In two dimensions, the subspaces are only three kinds. The single point at the origin, a line through the origin, and the whole plane. A line that misses the origin fails. A quarter of the plane fails too: scale a member by −1 and it leaves. Try them below.

Is it a subspace?

Pick a shaded set. Two teal points a and b sit inside it. The device checks the zero vector, the sum a + b, and the scaled copy −a.

Each set is a toy in the plane. The line through 0 is y = x/2; the offset line is y = x/2 + 1; the quadrant is x, y ≥ 0; the disk has radius 1.5 around the point (1, 1).

Now return to the figure. The shaded patches are small blobs with edges. Scale a member up by a large number and it leaves the blob. So, strictly, they are not subspaces. The paper calls them optimal subspaces: the small region of firing rates where activity should land before the reach. The lecture poses the question to make us notice the difference between a bounded region and a true subspace.

Rank: how many independent directions

The next warm-up asks for the rank of an outer product, a column vector times a row vector, abT. Take a = [1, 2] and b = [3, 4]. The product is the 2 by 2 table with rows [3, 4] and [6, 8]. The second row is exactly twice the first.

The reason is general. Every column of abT is a rescaled copy of a. Any two columns are therefore dependent: one is the other times a number. The rank of a matrix is the number of columns you cannot build from the others. Here that is 1. The lecture’s answer is the same: every column is a rescaled version of a, so any set of two or more is linearly dependent.

For neurons, rank 1 means every neuron follows one shared pattern in time, each at its own scale. Chapter 8 returns to that exact picture.

Orthogonal matrices

A square matrix is orthogonal when its columns are at right angles to each other and each has length 1. Such a matrix turns space without stretching it. The warm-up asks about the matrix with rows [1, 1] and [−1, 1].

Check. The columns are [1, −1] and [1, 1]. Their dot product is 1 − 1 = 0, so they are orthogonal. Each has length √2, not 1. So the matrix is not orthogonal in the full sense, although its columns are. Divide every entry by √2 and it becomes a pure 45 degree turn.

The null space

The last warm-up asks for a vector in the null space of [1, 3]. That is the set of vectors the matrix sends to zero. We need x with 1·x1 + 3·x2 = 0. Try [−3, 1]: −3 + 3 = 0. It works, and so does every multiple of it. The zero vector also works, which fits the lecture’s answer, [−3, 1] or [0, 0].

Read it as information. Feed [−3, 1] to this matrix and you get 0. Feed it twice that and you also get 0. The matrix cannot tell these inputs apart from silence. It destroys that direction. Chapter 5 turns this into the key fact about inverses.

Which of these sets of points in the plane is a subspace?

Chapter 3

The SVD as Three Steps

See any matrix as a rotation, then a stretch along the axes, then another rotation

Take two neurons and feed their rates into a matrix A. The output is two new numbers, say two readouts that a decoder computes. For a given input pair, the output is A times the input. Now ask: what does A do to every possible input at once?

Draw all the inputs of length 1. They form a circle. Push each one through A. The circle comes out as an ellipse, tilted and squashed. The ellipse is the whole story of the matrix, and the SVD reads it off. Its long axis tells you the direction A amplifies most. Its short axis tells you the direction it amplifies least.

The claim

The lecture introduced the SVD on the whiteboard, then recapped it on slides. The statement is short. Any matrix A can be written as the product of three matrices:

A
Any matrix, of any shape. In this chapter, a 2 by 2 matrix that maps two inputs to two outputs.
U
An orthogonal matrix. Its columns u1, u2, … are the left singular vectors: the directions on the output side. They have length 1 and meet at right angles, so UTU = UUT = I.
S
A diagonal matrix: zeros everywhere except the main diagonal. Its entries s1 ≥ s2 ≥ … ≥ 0 are the singular values. They are never negative, and by convention are listed from big to small.
VT
The transpose (rows and columns swapped) of an orthogonal matrix V. The columns of V, v1, v2, …, are the right singular vectors: the directions on the input side.

These are the exact pieces the lecture labels on its handwritten slides: left singular vectors in red, singular values in blue, right singular vectors in green. We borrow the colour roles in our own picture: purple for the left vectors, warm for the singular values and pink for the right vectors.

Why “rotate, stretch, rotate”

Read the product from the right, because that is the order a vector meets the matrices. A vector x meets VT first. This step is a rotation (possibly with a flip of an axis). It does not change the length of x. It only turns the input so that the right singular vectors line up with the coordinate axes.

Next x meets S. The matrix S is diagonal, so it simply multiplies the first coordinate by s1, the second by s2, and so on. This is the stretch along the cardinal axes: the only step that changes lengths. Last comes U, a second rotation, which carries the stretched vector into the output space.

The lecture draws it as a chain of four circles: x, then VTx, then SVTx, then y = USVTx, with the arrows labelled rotate, stretch, rotate.

A worked example

Take A with rows [3, 0] and [4, 5]. Its SVD is, with every step checked:

Test the first pair. Av1 = [3, 9]/√2 ≈ [2.12, 6.36]. Its length is √(4.5 + 40.5) = √45 = s1, and its direction is [1, 3], which is u1. So Av1 = s1u1. The same holds for the second pair: Av2 = [3, −1]/√2 has length √5 and points along u2.

That is the cleanest way to hold the SVD in your head: A sends the input direction vi to the output direction ui, stretched by si. Try it below. The slider walks the unit circle through the three steps.

Rotate, stretch, rotate

The dots start on the unit circle. The pink lines are the right singular vectors. Drag the stage from 0 to 3 and watch them turn, stretch, and turn again into s1u1 and s2u2. Change the two stretches to see a different matrix.

0.0
6.7
2.2

Illustrative sliders. The two turn angles are fixed at 45° (input side) and 71.6° (output side), the angles of the worked example, whose stretches are 6.7 and 2.2. The result is exactly A = U S VT for the matrix printed in the read-out.

At stage 3 the circle has become an ellipse whose axes are exactly s1u1 and s2u2. Set s2 to 0. The ellipse collapses to a line. You have found a matrix that loses a whole direction. That will matter in Chapter 5.

When a vector x goes through A = USVT, which step is the only one that can change its length?

Chapter 4

Working With the Three Factors

Use A = USVT to build the inverse, and to find the SVD of A times its own transpose

You are handed the three factors of a matrix and nothing else. No entries, only U, S and V. The lecture gives three small problems that you can solve from these alone. Doing them is the best way to learn how the factors behave. Each one uses just two facts, so we state them first.

Two facts to carry

Fact one. U and V are orthogonal, so their transposes undo them: UTU = I and VTV = I. Here I is the identity matrix, which changes nothing. A rotation turned back by its own transpose gets you home.

Fact two. The transpose of a product reverses the order: (AB)T = BTAT. The lecture prints this as a reminder beside problem 2. Also, S is diagonal, so ST = S, and multiplying two diagonal matrices just multiplies their entries.

Problem 1: the inverse

The question: write A−1 in terms of U, S and V, and show that AA−1 = I. The guess is A−1 = V S−1 UT. Here S−1 is the diagonal matrix with entries 1/si. Multiply it out, one cancellation per line:

USVT
The matrix A, written with its three factors.
V S−1 UT
The proposed inverse. The order is reversed and each factor is undone: U turned back by UT, S undone by 1/s, V undone by itself.
VTV
Meets in the middle and becomes I, by fact one.
S S−1
Each entry is s × (1/s) = 1, so this is I as well. Then U UT = I closes the proof.

The picture is simple: to undo a rotate, stretch, rotate, you un-rotate, un-stretch and un-rotate in the opposite order. Test it on our example, the matrix with rows [3, 0] and [4, 5]. Its determinant is 15, and its inverse has rows [1/3, 0] and [−4/15, 1/5]. Multiply the second row of A by the first column of the inverse: 4·(1/3) + 5·(−4/15) = 4/3 − 4/3 = 0. The second row by the second column: 5·(1/5) = 1. The inverse works.

Problem 2: A times its transpose

Now take AAT. Use fact two to write AT = (USVT)T = V S UT. Then:

AAT = (USVT)(VSUT) = US(VTV)SUT = US2UT

That last form is itself an SVD. It has the same left singular vectors U on both sides, and the singular values are now the originals squared, si2. The lecture’s slide states exactly these two bullet points.

Problem 3: the transpose times A

The same move gives ATA = (VSUT)(USVT) = V S2 VT. The roles flip. Now V appears on both sides and the squared values sit in the middle. The check on our example: ATA has rows [25, 20] and [20, 25]. Its trace (sum of the diagonal) is 50, which equals 45 + 5, the squares of 6.708 and 2.236. Its determinant is 625 − 400 = 225 = 45×5. So the squared singular values 45 and 5 are indeed what we find.

Check the identities

Set the four entries of A. The device computes its SVD, then separately finds the eigenvalues of ATA and of AAT. The three sets of bars should agree.

3
0
4
5

Exact for any 2 by 2 matrix. The singular values come from a Jacobi SVD; the eigenvalues of ATA and AAT come from the closed formula for a symmetric 2 by 2 matrix, so the two routes are independent.

What these results tell us

A matrix of the form ATA is symmetric. An eigenvector of a square matrix is a direction it only stretches and never turns: Ax = λx, where the stretch λ is the eigenvalue. Our matrix is also positive semi-definite (PSD): every eigenvalue is zero or larger. The lecture notes point out that covariance matrices are always of this type, which is why they matter so much in the course. For such a matrix the SVD is also an eigendecomposition: ATA = V S2 VT, with the eigenvectors in V and the eigenvalues si2. This is the link to eigenvalues and PCA, taken up in PCA and eigenvalues.

One more consequence from the notes: the absolute value of the determinant is the product of the singular values. Only S changes volume; the two rotations do not. For our example that is 6.708 × 2.236 = 15.

A has singular values 3 and 2. What are the singular values of ATA?

Chapter 5

Undoing a Matrix, and What to Do When You Cannot

Run the three steps backwards for the inverse, then learn the pseudo-inverse for a matrix that destroys a direction

A decoder maps the activity of two neurons to a cursor position: y = Ax. Later you see only the cursor and want to know what the neurons did. You want to run the map backwards. That is the job of the inverse matrix, written A−1. It is the matrix with A−1y = x for every x.

The lecture’s slides show the idea as a round trip. The forward arrows read VT, then S, then U, taking x to y. Beneath them, the backward arrows read UT, then S−1, then V, taking y back to x. No new idea is needed. You simply undo the last step first.

The inverse, step by step

Start from y = USVTx. First undo the last rotation. Multiply by UT, which is U’s inverse, giving SVTx. Second, undo the stretch. Multiply each coordinate by 1/si, which is the matrix S−1. Third, undo the first rotation by multiplying by V. You get x back. Written as one matrix:

A−1
The inverse of A, if it exists.
V
Un-rotates the input side, the last step of the round trip.
S−1
Diagonal, with entries 1/s1, 1/s2, … Each one cancels a stretch.
UT
Undoes the output rotation, the first step of the round trip.

Chapter 4 proved the product works. The lecture’s own check on slide 20 is the other order, A−1A = (VS−1UT)(USVT) = VS−1(UTU)SVT = V(S−1S)VT = VVT = I, and it leaves the other order, AA−1, as an exercise.

A worked case. Our example A (rows [3, 0] and [4, 5]) sends x = [1, 0] to y = [3, 4]. Its inverse, with rows [1/3, 0] and [−4/15, 1/5], sends [3, 4] to [1/3·3, −4/15·3 + 4/5] = [1, −4/5 + 4/5] = [1, 0]. The input is recovered exactly.

When a stretch is zero

Look at S−1 again. It needs 1/si. If some singular value is exactly 0, that entry would be 1/0, which does not exist. So no inverse exists. The SVD makes the reason visible: a zero stretch squashes a whole direction to nothing.

Say the second singular value is 0. Then any input along v2 is multiplied by zero and vanishes. Two different inputs that differ only along v2 give the very same output. Given the output, you cannot tell which input made it. The information is gone. This is the null space from Chapter 2, seen from the SVD: the null space is spanned by the right singular vectors whose stretch is zero.

The pseudo-inverse

You cannot recover what was destroyed. But you can recover everything that survived. The matrix that does this is the pseudo-inverse, written A†. The recipe is one line: invert every singular value that is not zero, and leave every zero where it is.

A†
The pseudo-inverse. It equals A−1 when A is invertible.
S†
Diagonal, with 1/s1, …, 1/sk for the k non-zero singular values, and zeros after them.

Try it on the smallest case. Take A with rows [1, 2] and [2, 4]. It is rank 1, with s1 = 5 and s2 = 0. Feed it x = [1, 0]. The output is [1, 2]. The pseudo-inverse is the matrix with rows [1, 2] and [2, 4], divided by 25. Apply it to [1, 2]: [(1 + 4)/25, (2 + 8)/25] = [0.2, 0.4]. That is not [1, 0]. It is the projection of [1, 0] onto the one direction that survived, the line through [1, 2]. The best you can do is recover the part that was not destroyed.

Now play with it. The device uses the example matrix and an input vector. Lower the second stretch toward zero and choose which inverse to use.

Undo the matrix

The blue arrow is the input x = [1, 0]. A maps it to y, and the chosen inverse tries to bring it back (green arrow). The pink lines are the right singular vectors. Lower s2 to 0.

2.2

Illustrative: s1 = 6.7 is fixed, the singular vectors are those of the worked example, and x = [1, 0] has length 1. In exact arithmetic the plain inverse works for any s2 above 0; Chapter 7 shows why small values are still trouble.

At s2 = 0 the plain inverse fails, because it needs 1/0. The pseudo-inverse returns [0.5, 0.5], the shadow of x on v1. It lost exactly the part of x along v2, a length of 0.71.

A has singular values 4, 2 and 0. How do you build its pseudo-inverse from the SVD?

Chapter 6

Rank, and the Shapes of Tall and Wide Tables

Count the independent directions of a matrix by counting its non-zero singular values, then fit the SVD to a recording that is not square

A lab records 100 neurons for 500 time bins. The table has 100 rows and 500 columns. Is every one of those 500 columns new information? Or do the neurons only ever combine a handful of patterns, so most columns can be rebuilt from a few others? The answer is the rank of the table, and the SVD gives it by counting.

Rank is a count of stretches

Chapter 2 said the rank is the number of independent columns, and that it equals the number of independent rows. The two always agree, which is a small miracle of linear algebra. For an m by n matrix, the rank can never exceed the smaller of m and n.

The SVD gives a much easier recipe. The lecture states it on a single slide: the rank of A is the number of non-zero singular values. If s1, …, sk are above zero and sk+1, …, sn are zero, then rank = k. The slide draws S with a red diagonal of live entries and zeros below.

The reason follows from the three steps. Each non-zero stretch keeps one output direction alive. Each zero stretch kills one. The number of surviving directions is the dimension of what A can produce, which is the rank. The earlier rank-1 example agrees: the matrix with rows [1, 2] and [2, 4] had stretches 5 and 0, so rank 1.

Three named spaces, read straight off the SVD

The lecture notes go one step further. Suppose A has rank k. Then the SVD hands you an orthonormal basis (a set of right-angle, length-1 directions) for three spaces at once:

The notes prove the null-space claim, and the proof is short enough to follow. Take an input x that is a mix of the last n − k columns of V. Then Ax is the same mix of the vectors Avi (linearity). Now Avi = US(VTvi). Since V is orthogonal, VTvi is a vector with a single 1, in position i, and zeros elsewhere. Then S times that vector is si in position i. For i above k, si = 0, so the product is zero. Every term vanishes, and Ax = 0.

So the null space has dimension n − k. Our 100 by 500 table, if its rank were 3, would erase a 497-dimensional space of time patterns. That is why a recording of a few patterns is so compressible.

Tall and wide matrices

Real recordings are almost never square. The lecture’s slide 25 shows both shapes. A tall, skinny matrix has more rows than columns, like many neurons and few time bins. A short, fat matrix has more columns than rows, like our 100 by 500.

For an m by n matrix, U is m by m, V is n by n, and S is m by n, so S is not square. Its diagonal runs only as far as the smaller side, so it holds at most min(m, n) singular values. A matrix with these shapes cannot be invertible, but the pseudo-inverse still exists by the same formula.

Look at S for 100 by 500. It has 100 rows and 500 columns. Only the first 100 columns have a diagonal entry. The other 400 columns are all zeros. Those zeros are multiplied against the last 400 rows of VT, so those rows of VT never affect A. The lecture colours them grey and writes “irrelevant”. Drop them and you get the reduced SVD: U stays 100 by 100, S becomes 100 by 100, and VT becomes 100 by 500. Same matrix, a fifth of the storage in V.

Shape the factors

Choose a recording size and a rank. Boxes are drawn to scale. Grey parts never touch A. Switch to the reduced SVD to drop them.

100
500
3

Sizes are chosen by you; the counts are exact. The rank slider is capped at the smaller side, and a rank of 8 or more is capped to that limit for small recordings. Real recordings are rarely exactly low rank; Chapter 8 handles the approximate case.

Try 100 neurons by 500 bins with rank 3. The read-out counts the numbers in each factor. The full V has 250,000 numbers, and the reduced VT has 50,000. Then try the other shape, 8 neurons by 20 bins, to see how the roles swap.

A is a 100 by 500 matrix with exactly 3 non-zero singular values. What is the dimension of its null space?

Chapter 7

True and Practical Non-Invertibility

Learn why a matrix with a tiny stretch is invertible on paper and useless in a computer, and how one ratio measures it

Two neurons in a motor area are almost twins. Their firing rates rise and fall together, with only a faint difference. You build a decoder that must tell them apart. On paper, the matrix is fine: the second singular value is not zero, so an inverse exists. In the lab, the decoder returns wild answers. What went wrong?

A spectrum that drops off a cliff

Chapter 5 said a zero stretch destroys a direction, so no inverse exists. That is true non-invertibility. But real matrices rarely hit exactly zero. Instead they have a few big singular values, then a slide into very small ones. The lecture draws this as a plot of singular value against its number: the first dot is high, the second is lower, and the rest lie almost on the axis.

A tiny singular value is not zero. The inverse needs 1/si, and 1/(a tiny number) is a huge number. Any small error that lands along that direction is multiplied by the huge number and comes back enormous. This is practical non-invertibility: the inverse exists, but using it on noisy numbers is hopeless.

The condition number

One ratio measures the danger. The condition number is the largest singular value divided by the smallest:

cond(A)
The condition number. It is at least 1, and equals 1 only for a matrix that stretches every direction equally, such as a pure rotation.
s1
The largest singular value: the most the matrix ever stretches a unit vector.
sn
The smallest singular value, the least it ever stretches one. It sits at the bottom of the lecture’s plot.

The ratio matters, not either number alone. Multiply a matrix by 1,000 and every singular value grows by 1,000, but the ratio stays the same. The matrix is no easier to invert. What counts is how lopsided the stretching is.

Where the error comes from

A worked case shows the mechanism. Let s1 = 1 and s2 = 10−6, so the condition number is 106. Send an input along the strong direction: x = v1. The output is y = u1, with length 1.

Now suppose your measurement of y carries 0.1% noise, a number as small as 10−3, and the noise falls along u2. Undo the matrix. The noise passes through S−1, which multiplies it by 1/s2 = 106. The recovered input is off by 10−3 × 106 = 1,000. The true input had length 1. The error is a thousand times bigger than the answer.

The general rule: a relative error in the output can grow by up to the condition number in the recovered input. Even with perfect data, a computer rounds every number to about 16 digits (double precision). A condition number of 1010 means you can lose about 10 of those 16 digits, leaving six.

Invert a nearly singular matrix

The curve shows how big the error in the recovered input is for each condition number. The teal marker is your matrix. Choose the noise in the measured output.

106

Computed live in your browser in 64-bit floating point: a 2 by 2 matrix with s1 = 1 and s2 = 1/condition, input along v1, noise along u2, inverted with the closed 2 by 2 formula. Relative error means error length divided by input length (1). The two dashed lines mark the lecture’s 1012 and the notes’ 1014.

How big is too big?

The slide gives a rule of thumb: a matrix is not practically invertible if its condition number is above about 1012. The notes put the double-precision limit nearer 1014. The two numbers differ because the real limit depends on how much accuracy you need. Rounding alone eats 16 digits at 1016, and a useful answer needs a few digits left. Such a matrix is called ill-conditioned or, loosely, singular. A numerical library will often warn you: “matrix is close to singular”. In Python, numpy.linalg.cond(A) returns it.

A common fix uses the pseudo-inverse from Chapter 5. Decide a cut-off. Treat every singular value below it as zero and leave it out of S†. You lose a tiny sliver of the signal and avoid the huge amplification. This is also the first step towards the approximation of the next chapter.

A matrix has singular values 200, 40 and 0.0002. Roughly what is its condition number, and what does it tell you?

Chapter 8

Sums of Outer Products and the Best Simple Copy

Rewrite the SVD as a stack of rank-1 pieces, then keep only the strongest few to approximate a matrix

You photograph a recording as a heat map: neurons down the side, time across the top. A friend asks for a copy, but the email limit allows only a few hundred numbers. You could crop it, or blur it. The SVD offers something smarter: send the pieces that carry the most, and drop the pieces that carry the least.

The same SVD, written as a sum

So far we multiplied three matrices. There is an equivalent way to read the product. Take column i of U, call it ui. Take column i of V, call it vi. Multiply the column by the row, uiviT. That product is an outer product, a full table of numbers built from just two vectors. From Chapter 2 we know it has rank 1.

Then the SVD says that A is a sum of these pieces, each scaled by its singular value:

A
The whole matrix, for example a neurons by time table.
s1 u1 v1T
The strongest piece. A column u1 (how each neuron takes part) times a row v1T (the pattern in time), scaled by s1. It is a rank-1 matrix.
sn un vnT
The weakest piece. Because the singular values are sorted, each piece adds less than the one before.

The lecture colours each piece differently: red, blue, green, and so on, drawn as a stack of coloured squares joined by plus signs. A hand-worked example makes it concrete. Our matrix A has rows [3, 0] and [4, 5]. Its first piece is s1u1v1T = 1.5 × [[1, 1], [3, 3]] = [[1.5, 1.5], [4.5, 4.5]]. Its second piece is s2u2v2T = 0.5 × [[3, −3], [−1, 1]] = [[1.5, −1.5], [−0.5, 0.5]]. Add them: [[3, 0], [4, 5]]. The matrix returns exactly.

Keep the strongest pieces

Now truncate. Stop the sum after k terms and call the result A(k). The lecture states the key fact: the best rank-k approximation to A, in squared error, is the SVD truncated after k terms. No other matrix of rank k comes closer. For k = 1 this is the best separable approximation: one pattern in time multiplied by one pattern across neurons.

What does “closer” mean? The notes define the Frobenius norm: square every entry of the matrix, add them up, take the square root. It is the ordinary vector length, applied to the table laid out as one long list. And there is a neat result: the squared Frobenius norm equals the sum of the squared singular values.

Check on our example. The squared entries of A are 9 + 0 + 16 + 25 = 50. The squared singular values are 45 + 5 = 50. They match. The proof in the notes is two lines: the squared norm is the trace (sum of the diagonal) of ATA, which is the trace of V S2 VT, which, because a trace does not change when you cycle the factors, is the trace of S2.

It follows that the error from dropping pieces is clean. If you keep k pieces, the squared error is exactly the sum of the squares of the singular values you dropped. In our example, keeping one piece leaves the error [[1.5, −1.5], [−0.5, 0.5]] with squared size 2.25 + 2.25 + 0.25 + 0.25 = 5, which equals the dropped s22 = 5.

Fraction of variance accounted for

This gives a score for any k. The lecture calls it the fraction of variance accounted for:

F(k)
The share of the matrix’s squared size kept by the first k pieces, from 0 to 1.
s12 + … + sk2
The sum of squared first k singular values: what you keep.
s12 + … + sN2
The sum over all N singular values: the whole squared size.

For our example, F(1) = 45/50 = 90%. One piece rebuilds 90% of the matrix’s squared size. Note the word “variance”. It is a true variance only when each row has had its average removed first, as the toy recording below does. Without that, the first piece may only be capturing the averages.

Stack the pieces

Top row: the first three rank-1 pieces siuiviT. Below: the sum of the first k pieces, the recording, and the error left over. Drag k.

2

The same toy recording as the hero: 16 neurons by 48 bins, rows centred. Errors are measured directly on the matrices, and then compared with the sum of dropped squared singular values: they agree.

Drag to k = 3 and the error tile turns almost flat. The rest is the toy’s noise. That is the practical use: low rank splits signal from noise, if the signal is built from a few patterns.

A has singular values 6, 3 and 1. You keep only the first piece. What fraction of the squared size is kept?

Chapter 9

SVD on a Population of Neurons

Split a neurons by time table into timecourses and neural weights, and see how many patterns the population really uses

Twelve neurons are recorded while an animal prepares a movement. Each neuron has its own rate trace across 60 time bins. At first glance they look different: one rises early, another dips, a third does both. But perhaps they are all echoes of a few shared signals, each neuron listening to them with its own volume. If so, 12 messy traces hide something simple.

The lecture’s picture

The last content slide of the lecture draws this. A table A sits on the left. Its rows are neurons and its columns are time, and each row is a small firing-rate curve. An approximation sign follows, then a sum of two terms. The first term pairs a column of numbers, labelled neural weights #1 (s1u1), with a curve labelled timecourse #1 (v1T). The second term does the same with #2.

In symbols this is Chapter 8’s truncated sum, A ≈ s1u1v1T + s2u2v2T, with a meaning attached to each part:

Read this as a model of the population. Neuron j’s trace is s1u1(j)·v1 + s2u2(j)·v2. Every neuron mixes the same two timecourses. Only the mixing weights differ.

The data flow, with shapes

With N neurons and T time bins, A has shape N × T. Our example has N = 12 and T = 60. The reduced SVD returns U with shape 12 × 12, s with 12 numbers and VT with shape 12 × 60. Keep the first two columns of U and the first two rows of VT. Then 12 × 2 weights times 2 × 60 timecourses rebuilds a 12 × 60 table. That is 2 × (12 + 60) = 144 numbers in place of 720.

A subspace view ties back to Chapter 2. Each neuron is an axis in a 12-dimensional space, and the population’s state at one time bin is one point there. The left singular vectors u1 and u2 span a flat two-dimensional sheet through the origin of that space. If the table is nearly rank 2, every point of the population’s trajectory lies close to that sheet. This is the same shape of idea as the premotor figure in Chapter 2, and the starting point of the neural manifolds lesson.

How many patterns?

The singular values answer that. Plot them in order. A few tall bars followed by a floor of small ones says the table is built from a few patterns and noise. A slow slide with no clear drop says there is no short list. Chapter 8’s fraction of variance turns the plot into one number.

Try it. The device builds a toy population from exactly two hidden patterns, adds noise, and runs the SVD. It then shows what comes out.

Find the hidden patterns

Top: the noisy population, 12 neurons by 60 bins. Middle: the two recovered timecourses (pink) against the true ones (dashed). Bottom: the neural weights (purple). Raise the noise.

12

A toy, not real data: pattern 1 is a bump, pattern 2 is its slope, neuron weights are a cosine and a sine across the 12 neurons, strengths 30 and 20, plus Gaussian noise of the chosen size. Rows are not centred. The sign of a singular vector pair is arbitrary, so the recovered curves are flipped to match the true ones.

At low noise the two timecourses come back almost exactly. As noise rises, the third singular value climbs toward the second, and the recovered curves wobble. Past a point the second pattern drowns. SVD does not know there are two hidden patterns. It reports all 12 and lets you read the drop.

Cautions that carry to real data

Three cautions come with this tool. First, singular vectors are orthogonal by construction. Real neural patterns need not be. SVD gives the best orthogonal set, which may mix true patterns. Second, remove each neuron’s average first if you care about change over time. Otherwise the first piece often just encodes the averages. With that step, SVD is principal component analysis, taught in PCA. Third, the choice of k is a judgement. Look for the drop in the spectrum, and check the result on held-out data.

You run the SVD of a 12 neurons by 60 bins table. The first two singular values are large and the rest are tiny. What do v1 and u1 tell you?

Chapter 10

The Field Guide

Keep the numbers, the checklist and the code, and find the lessons that build on them

The lecture closes on a one-slide summary: the SVD, the inverse, the pseudo-inverse, rank, non-square matrices, the condition number, ill-conditioned matrices and low-rank approximation. All of them are one fact read from different angles. Write any matrix as A = USVT: turn, stretch along the axes, turn. Then look at the stretches.

The numbers and terms

TermWhat it isHow the SVD gives itOur example
Singular valuesHow strongly A stretches each directionThe diagonal of S, sorted, never negative6.708 and 2.236
InverseThe matrix that undoes AA−1 = V S−1 UTRows [1/3, 0] and [−4/15, 1/5]
Pseudo-inverseThe best undo when some stretch is zeroInvert the non-zero s, keep the zeros[1, 2; 2, 4] gives [0.04, 0.08; 0.08, 0.16]
RankNumber of independent directionsCount of non-zero singular values2 (and 1 for [1, 2; 2, 4])
Condition numberHow lopsided the stretching iss1 / sn; trouble above about 1012 (slide) to 1014 (notes)6.708 / 2.236 = 3
ATA and AATSymmetric, non-negative matricesV S2 VT and U S2 UTEigenvalues 45 and 5
DeterminantHow A scales volume|det A| = product of singular values6.708 × 2.236 = 15
Squared sizeFrobenius norm squaredSum of squared singular values9 + 0 + 16 + 25 = 50 = 45 + 5
Best rank-k copyClosest matrix of rank kKeep the first k terms of the sumk = 1 keeps 45/50 = 90%

A checklist for using the SVD

  1. Lay out the table. Rows are neurons, columns are time bins. Decide the bin width, then remove each neuron’s average if you care about change.
  2. Compute the SVD with the reduced form, so the factors are small.
  3. Read the spectrum. Look for a drop between large and small singular values. That says how many patterns the table holds.
  4. Check the condition number before you invert anything. If it is huge, use the pseudo-inverse with a cut-off.
  5. Truncate to k pieces and report the fraction of squared size kept.
  6. Interpret with care. The vectors are orthogonal by construction. Name a pattern only with outside evidence.

The core as code

pythonimport numpy as np

A = np.array([[3., 0.], [4., 5.]])
U, s, Vt = np.linalg.svd(A)            # s = [6.708, 2.236], Vt is V transposed
assert np.allclose(U @ np.diag(s) @ Vt, A)

A_inv = Vt.T @ np.diag(1 / s) @ U.T   # V S^-1 U^T
print(np.linalg.cond(A))               # 3.0 = s1 / s2
print(np.linalg.matrix_rank(A))        # 2 non-zero singular values
print(np.linalg.pinv(np.array([[1., 2.], [2., 4.]])))  # [[.04 .08] [.08 .16]]

# a neurons-by-time recording: shapes (N, T) = (100, 500)
rng = np.random.default_rng(0)
X = rng.normal(size=(100, 500))            # stand-in for real rates
Xc = X - X.mean(axis=1, keepdims=True)  # remove each neuron's average
U, s, Vt = np.linalg.svd(Xc, full_matrices=False)  # (100,100) (100,) (100,500)
k = 3
Ak = (U[:, :k] * s[:k]) @ Vt[:k]       # best rank-3 copy, shape (100, 500)
kept = (s[:k]**2).sum() / (s**2).sum()   # fraction of squared size kept
assert np.isclose(np.linalg.norm(Xc - Ak)**2, (s[k:]**2).sum())

The last line is Chapter 8 in code: the squared error equals the sum of the squares of the singular values you dropped.

Where this sits, and the way on

SVD is the working language of the rest of the course and of our brain-computer interface series. Eigenvalues and PCA are the same maths seen on a covariance matrix: see eigenvalues and PCA. The idea of a few patterns behind many neurons returns in neural manifolds, in linear decoders (where the pseudo-inverse and the condition number decide whether a decoder works) and in latent variable models. The next lecture, principal components analysis, builds directly on this one.

Credit. This lesson follows Lecture 2 of NEU 560, Statistical Modeling and Analysis of Neural Data, Princeton University, Spring 2024, taught by Jonathan Pillow. The slides (PDF) and the lecture notes on SVD are on the course site. The premotor figure on slide 4 is from Churchland, Yu, Ryu, Santhanam and Shenoy, Neural variability in premotor cortex provides a signature of motor preparation, J Neurosci, 2006. All examples, numbers marked as toys, and devices here are our own.

The takeaway. Every matrix is a rotation, a stretch and a rotation. The stretches tell you the rank, the condition and the best simple copy. In a neural recording, the few big stretches are the few patterns the population really uses.

Now press Present or Teach and explain, from memory, what the three factors do to a vector, why a zero stretch has no inverse, why a tiny stretch is almost as bad, and how a rank-2 copy of a recording is built. Then go back to the recording at the top, press 1 piece and 3 pieces, and say which singular value each step added.

You have the SVD of a 100 by 500 recording and want a compressed copy that keeps as much as possible. What do you do?
NEU 560 Lecture 2: SVD and Its Applications
Course site · Back to Gleams