Least Squares

A neuron fires more as contrast rises. Choose the line that predicts its spike counts with the smallest total squared miss, and see how one formula finds it.

Finish PCA with its ellipse, its centering and its row and column twins. Then meet linearity and a fit used all over neuroscience, with two proofs.

Drag the slope and watch the yellow squares: their total area is the error. Then press Solve with the formula. After that we build every piece, from the PCA ellipse to the right angle that makes the formula work.

You need the idea of a vector (a list of numbers) and a rough memory of slopes. We build the rest from zero.

Based on Lecture 4 of Princeton NEU 560, Statistical Modeling and Analysis of Neural Data (Spring 2024), taught by Jonathan Pillow, and its lecture notes on least squares: the slides (PDF) and the notes are on the course site. Every idea is re-taught here in our own words and devices. It follows Lecture 3, PCA.

Fit a line to spike counts

2.00

Illustrative data: 14 made-up trials of one neuron, seeded so they are the same on every visit. The line has no intercept, so it passes through zero spikes at zero contrast. Each yellow square has a side equal to the miss, so its area is that trial's squared error. The solve button computes the slope as the sum of x times y divided by the sum of x squared, the one-column case of the lecture's formula.

Chapter 0

Why a Neuroscientist Fits Lines

See how finishing PCA and learning least squares give you two tools that every later lecture of the course leans on

Picture a recording room. A monkey watches a screen, and a thin wire sits next to one neuron in its visual cortex. The screen shows a pattern at five contrast levels, over and over. On every trial you write down two numbers: the contrast, and how many spikes (brief electrical pulses, the neuron’s only way of speaking) the neuron fired in one second.

After a hundred trials you have a hundred pairs. A pattern is easy to see: higher contrast, more spikes. Now you want more than a feeling. You want a rule that takes a contrast and returns a predicted spike count, and you want the best rule of its kind. Finding that rule is called regression, and the simplest version, least squares, is the second half of this lecture.

What does “best” mean? Say a rule predicts 4 spikes on one trial and the neuron fires 7. The miss is 3. On another trial the rule predicts 6 and the neuron fires 4, so the miss is −2. Square each miss to get 9 and 4, and add them: 13. (These toy numbers are illustrative.) A rule with a smaller total is a better rule. Least squares picks the rule with the smallest total over all your trials. Squaring does two jobs. A miss up and a miss down cannot cancel, and a big miss counts for far more than a small one. Chapter 7 does this with full arithmetic.

Both halves start from one object: a table with one row per trial or time bin and one column per neuron. PCA reads the whole table and asks which directions carry its spread. Least squares reads the same table as inputs, adds one column of outputs, and asks which weights predict that column. Keep that picture in mind. It makes the two tools easy to tell apart.

The first half finishes a job started in Lecture 3. Now suppose you record not one neuron but a hundred at once. Each moment gives a list of a hundred spike counts, and you cannot look at a hundred axes. Principal component analysis (PCA) finds the few directions along which the population actually moves, so you can draw it. Lecture 3 built PCA from the singular value decomposition. This lecture tidies the loose ends: what the ellipse is, why you centre the data, what a projection looks like, and how PCA treats neurons versus time.

The shape of the session

The course site lists this session as PCA II and least squares. Its slides run in two blocks. A short one-slide warmup opens the first block, and the green “PCA Leftovers” divider starts the real work. The second block begins at the “Least Squares (on board)” divider. Professor Pillow derives the regression solution on the whiteboard there, so the written lecture notes carry that part of the lesson. We follow both.

One paragraph of logistics, and then we leave it. The slides announce that the scores for the first quiz are posted on Canvas (the course’s grading site) and that the first homework, HW0, is on the course website. Nothing else in the session is administrative.

Where this shows up in brain-computer interfaces

This is a course spine for the brain-computer interface (BCI) series, so keep one picture in mind. A decoder reads a person’s intended hand velocity from the spike counts of many neurons. The simplest decoder is a weighted sum of those counts, and its weights are found by least squares (Linear Decoders builds it). PCA, in turn, shrinks a hundred noisy channels to a few clean ones first (Neural Manifolds). Both tools come back again and again in the later lectures.

Try the hero once more before reading on. The slope you chose is a single weight. The formula that finds the best weight is short, and by the end of Chapter 9 you will be able to derive it two ways and know when it fails.

The road

  1. PCA is an ellipse: the second moment matrix, its eigenvalues, and a warmup you can do in your head.
  2. Centering and projecting: why PCA runs on centred data, and what a projection looks like.
  3. Row PCA and column PCA: neurons versus time, and why the two spectra can differ.
  4. Two ways to say PCA: keep the most, or lose the least.
  5. Beyond PCA: ICA, NMF, tensor methods, kernel PCA, factor analysis.
  6. Linearity: superposition, matrices, and the trap of the affine function.
  7. Least squares: the question: pairs, a design matrix, and a bowl of error.
  8. Derivation one: the right angle: the geometry behind the formula.
  9. Derivation two: calculus: the same formula, and where it breaks.
  10. The Field Guide: numbers, terms, code and where to go next.
You record 100 neurons and want to draw the population on a flat page. You also want to predict one neuron’s spike count from a stimulus. Which tools fit which job?

Chapter 1

PCA Is an Ellipse

Turn a cloud of two-neuron counts into a tilted oval, and read the oval’s axes off one small matrix

Record two neurons for 150 one-second bins. In each bin neuron 1 fired some number of spikes and so did neuron 2. Plot one dot per bin, with neuron 1 on the horizontal axis and neuron 2 on the vertical axis. If the neurons were independent, the dots would form a round blob. They are not independent: they receive some of the same input, so they rise and fall together. The dots form a tilted oval.

The lecture’s summary slide says exactly this: PCA is equivalent to fitting an ellipse to your data. The long axis of the oval is the first direction PCA finds. The short axis is the second. This chapter shows how a small table of numbers gives you both.

The data matrix

Stack the data into a table called X. Each row is one bin: a list of d spike counts, one per neuron. There are N bins, so X has N rows and d columns. We write its shape as N × d. For our example N = 150 and d = 2. The slide says each row is the response of d neurons to a single stimulus, and the picture is the same.

The cloud does not have to fill all d dimensions. Suppose two neurons always fire the same number of spikes. Every dot then lies on the line neuron 1 = neuron 2, and the whole cloud lives in a one-dimensional subspace (a flat slice of the space, here a line through the origin). Real neurons are never that exact, but they often come close.

The second moment matrix

How do you measure the shape of a cloud with a few numbers? Multiply the counts in pairs and add them up over all bins. This gives the second moment matrix, written C, with one entry for every pair of neurons.

C
The second moment matrix, d × d. Entry (a, b) is the sum, over all bins, of neuron a’s count times neuron b’s count. The diagonal holds each neuron’s sum of squared counts.
XTX
The way to compute it: flip X on its side (XT, the transpose) and multiply. The shape is (d × N) times (N × d), which gives d × d.
U
A d × d table whose columns u1, u2, … are the principal components (PCs): unit-length directions, each at a right angle to the others.
S
A diagonal table of numbers s1 ≥ s2 ≥ …. Each s says how much of the data’s sum of squares lies along its PC.
UT
The transpose of U. Together, U S UT is the SVD (singular value decomposition) of the symmetric matrix C, as in Lecture 2.

If the SVD is new to you, here is the plain version. It splits a table into a set of perpendicular directions and one stretch number for each. For a table like C, these are the directions that C only stretches and never turns. Lecture 2 builds it step by step.

A note on names. The slide writes the diagonal numbers as s1, s2 and calls them singular values of C. The course then says (slide 14) that the variances are the squared singular values of X itself. We follow the slide: si is the sum of squares along PC i, and the singular values of X are the square roots of these numbers.

The slide also reads the ellipse off these numbers. The PCs are the major axes of the ellipse, and the s values set the lengths of the axes. If you draw the ellipse one standard deviation wide, its half-lengths are the square roots of si/N. That scaling is our own clarification, and the shape is what matters.

Drag the two sliders below. The correlation says how much the neurons share input. The gain makes neuron 2 louder. Watch the PC arrows follow the oval.

Fit the ellipse

Each blue dot is one time bin. The warm oval is the fitted ellipse and the arrows are the PCs. Change how much the neurons share input, then change neuron 2’s gain.

0.70
1.00

Toy data: 150 bins from two made-up neurons, seeded so they are the same on every visit, with the average count subtracted from each neuron (Chapter 2 explains why). The PCs come from the eigen-decomposition of XTX computed live in your browser. The ellipse is drawn two standard deviations wide.

With no shared input the oval turns round and the PCs have no favourite direction. With strong shared input the oval thins into a cigar and s1 swallows nearly everything. The gain tilts the cigar toward the louder neuron, because the PCs follow where the counts are large.

The warmup problem

The slides open with one small question. Take any vector z that mixes the first two PCs: z = a u1 + b u2. What is zTCz? Here is the whole answer, step by step.

  1. Apply C to z. The SVD says C ui = si ui: C only stretches each PC. So Cz = a s1 u1 + b s2 u2.
  2. Multiply by zT on the left. You get a u1T and b u2T times those two terms: four products.
  3. Use the right angles. u1Tu2 = 0 and uiTui = 1. Two of the four products vanish.
  4. Read the result. zTCz = a2 s1 + b2 s2.

s1 = 9, s2 = 1, a = 2, b = 1  ⇒  zTCz = 4 × 9 + 1 × 1 = 37

Why care? Because zTCz equals the length of Xz squared. It adds up the squared projection of every data point onto z. So the formula says how much of the data’s energy a direction catches: a squared share of s1 for the u1 part and a squared share of s2 for the u2 part. The first PC wins because s1 is the biggest number you can be multiplied by.

That check is the last sentence in the read-out above. It uses z = 2u1 + u2 and computes zTCz two ways: by the formula and by projecting all 150 dots. Both agree.

A cloud of two-neuron counts is a thin oval tilted at 45 degrees. What do PC 1 and the number s1 tell you?

Chapter 2

Centering and Projecting

Subtract the average first, so PCA describes the shape of the cloud and not how far it sits from zero

Two neurons in visual cortex have busy baselines. Neuron 1 averages 23 spikes per bin and neuron 2 averages 12. Their counts wobble a few spikes around those averages. Plot the bins and you see a small cloud, far from the origin, out in the upper right of the plot.

Ask PCA for the main direction of this cloud, using the second moment matrix C = XTX from Chapter 1. The answer is disappointing. The long axis of the fitted ellipse does not run along the cloud’s shape at all. It points from the origin straight at the cloud’s centre. The lecture shows exactly this on two slides titled “what is the dominant eigenvector of XTX?”.

Why the raw matrix looks at the wrong thing

Think of C as measuring how far the data sit from the origin along each direction. If the whole cloud is far away, then almost all of that distance lies along the line to the cloud’s centre. Any small wobble is tiny next to it. So the biggest direction is simply “toward where the cloud is”, and the shape is lost.

Try it with six bins of two-neuron counts. They are made up for this example. The counts (neuron 1, neuron 2) are (20, 16), (22, 10), (24, 14), (26, 8), (18, 12), (28, 12). The averages are 23 and 12. The raw matrix XTX is [[3244, 1636], [1636, 904]]. Its eigenvalues are 4085.3 and 62.7, and its first PC points at 27.2 degrees. The line from the origin to the average (23, 12) sits at 27.6 degrees. They almost agree, and the 62.7 says the data seem to have no second direction at all.

Centering

Centering means subtracting the average from every row: replace each xi by xi − x̄, where x̄ is the list of averages, one per neuron. The cloud keeps its shape but now sits on the origin.

C
Now a covariance matrix. Entry (a, b) says whether neuron a and neuron b rise above their averages at the same time.
1/N
Divides by the number of bins, so the numbers do not grow just because you recorded longer. The lecture slide writes this factor.
(X − x̄)
The data table with each neuron’s average subtracted from its column. It still has shape N × d.

On the six bins, centering gives (X − x̄)T(X − x̄) = [[70, −20], [−20, 40]]. Its eigenvalues are 80 and 30. The first PC now points at −26.6 degrees, down and to the right, which is the real tilt of the cloud: when neuron 1 is high, neuron 2 tends to be low. Dividing by N = 6 gives variances of 13.3 and 5.0.

The slide gives the rule in one line: in practice, we almost always do PCA on centered data. Drag the cloud away from zero below, and flip between the raw and the centred view.

Move the cloud off zero

The blue cloud is tilted down to the right. The dashed line runs from zero to the cloud’s centre. Raise the distance, then switch between raw and centred PCA.

7.0

Toy data: 120 bins of two made-up neurons with spreads of 2.2 and 1.0 counts, tilted 27 degrees down to the right, seeded for every visit. The centre sits along a line at 30 degrees above the horizontal axis. The PCs are computed live in your browser from XTX (raw) or from the centred matrix.

At distance 0 the two views agree. As the cloud moves away, the raw arrow swings round to face the cloud’s centre and stays there. The centred arrow never moves, because centering removes the distance before PCA looks.

Projecting onto the PCs

Now use the PCs to draw a many-neuron cloud on a flat page. Take the first two PCs, u1 and u2. For each bin, compute two dot products: how far the bin’s counts lie along u1, and how far along u2. The slide writes this as the pair (u1 · xi, u2 · xi). Doing it for every bin at once is one multiplication: X times the table [u1 u2], of shape (N × d) times (d × 2), giving N × 2. Each row is a point on your page.

A dot product multiplies two lists entry by entry and adds the results. For a unit-length direction it measures how far a point reaches along that direction. That is all a projection is.

The slide ends with a quiet warning: the projection may not look elliptical, although PCA cannot tell. PCA uses only averages and the covariance, so it cannot see curves or clumps. Two clouds with the same covariance get the same PCs, whatever their shapes. The device below projects three made-up neurons into the PC 1 and PC 2 plane. Pick a shape and read what the plot says.

Project onto PC 1 and PC 2

Three neurons make a cloud in 3-D. Choose its true shape, then look at the flat view PCA gives. The dashed oval is the ellipse PCA fits to the same covariance.

Toy data: 150 bins of three made-up neurons, centred, then rotated into a fixed random direction so no axis lines up with a neuron. The horseshoe is a half circle with a little noise. The bars show each PC’s share of the sum of squares.

The horseshoe is a path a population might trace as a stimulus turns. PCA draws it faithfully here, but the ellipse around it is a poor summary. Always look at the projection, not only the numbers.

You run PCA without centering on counts whose averages are 23 and 12 spikes. What does the first PC mostly describe?

Chapter 3

Row PCA and Column PCA

Ask two questions of one table, which neurons move together and which moments look alike, and see why centering changes the answers

An experimenter shows the same flash 200 times and records 8 neurons. She averages the spike counts over the 200 trials, bin by bin. The result is a table with one row per neuron and one column per time bin: 8 rows and 40 columns. Each row is that neuron’s average response over time, a curve called the PSTH (peri-stimulus time histogram: average firing against time since the flash).

Lecture 3 gave you PCA for a table whose rows are points. Here the table can be read two ways. Read each row as a point: 8 points, each living in a 40-dimensional space of time bins. Or read each column as a point: 40 points, each living in an 8-dimensional space of neurons. The slides ask: what is the relationship between SVD(X) and PCA(X)? The answer is that both readings are one decomposition seen from two sides.

One table, one SVD, two PCAs

Write the SVD of the table as X = U S VT. If X is N × T (N neurons, T time bins), then U is N × N, S is the diagonal table of singular values, and V is T × T. Keep only the top k terms and you get the slide’s picture: X is about the sum, over k terms, of sj uj vjT. Each term is a weight for every neuron (uj, a column of N numbers) times a temporal profile (vj, a curve of T numbers). The slide credits this picture to Williams et al., Neuron 2018.

Row PCA treats the rows as points. Its second moment matrix is XTX, of shape T × T. The slide shows that XTX = V S2 VT. So the PCs are the right singular vectors of X, vectors with T entries. The slide calls them the time PCs: they are curves over time that the neurons share.

Column PCA treats the columns as points. Its matrix is XXT, of shape N × N, and XXT = U S2 UT. The PCs are the left singular vectors, vectors with N entries: the neuron PCs, weights across the neurons. Both PCAs report the same singular values, squared, as their variances.

XTX = V S2 VT (T × T, time PCs)  |  XXT = U S2 UT (N × N, neuron PCs)

A tiny table makes it concrete. Let X have two neurons and three bins: row 1 is (4, 2, 0) and row 2 is (2, 1, 0). These two curves have the same shape, one just twice as loud. XXT = [[20, 10], [10, 5]] and XTX = [[20, 10, 0], [10, 5, 0], [0, 0, 0]]. Both have a top eigenvalue of 25 and nothing else. So the singular value of X is 5, because 5 squared is 25. Row PCA finds the shared curve, (2, 1, 0) scaled to length one. Column PCA finds the loudness pattern, (2, 1) scaled to length one.

What centering does to the two

The slide then adds a warning: these will no longer be the same if you zero-centre. The reason is that the two PCAs centre different things. Row PCA treats rows as points, so it subtracts the average point: the column average, one number per time bin. Column PCA treats columns as points, so it subtracts each neuron’s own average over time. These are different matrices. They have different singular values.

On the tiny table, subtracting each neuron’s average over time gives rows (2, 0, −2) and (1, 0, −1), and the top variance is 10. Subtracting each time bin’s average over neurons gives rows (1, 0.5, 0) and (−1, −0.5, 0), and the top variance is 2.5. The raw table said 25. Three tables, three answers.

Try it on 8 neurons and 40 bins. Switch between the raw table and the two centred versions, and watch the heat map, the curve and the four biggest variances.

Centre it two ways

Top: the table, one row per neuron, with warm for high and blue for below zero. Middle: the first time PC. Bottom: the four biggest variances, with the raw table as a faint outline. Choose how to centre.

Toy data: 8 made-up neurons with baselines from 5 to 20, a shared early bump and a slow ramp mixed in with different weights, plus noise, seeded for every visit. The bars on the left of the heat map are the first neuron PC. Variances are the squared singular values of the table shown, computed live. For one table, the neuron PCs and time PCs share their variances.

On the raw table, the top variance is huge because every neuron has a large baseline: the first PC mostly records “this neuron is loud”. Centering each neuron over time removes the baselines and leaves the response shapes. Centering each time bin over neurons removes what all neurons do together and leaves how they differ from one another. Neither is wrong. They answer different questions.

Which table to decompose

The last item of the lecture’s PCA summary compares SVD(X), SVD(XTX) and SVD(XXT). They are three routes to the same numbers. The SVD of X gives U, S and V together. The SVD of XTX gives V and S squared. The SVD of XXT gives U and S squared. Pick the smaller of the two matrices to decompose: for 8 neurons and 40 bins that is XXT, 8 × 8, not the 40 × 40 one.

NamePoints areMatrixPCs live inCentre by subtracting
Row PCA (time PCs)rows of X (neurons)XTX, T × Ttime, RTthe column average
Column PCA (neuron PCs)columns of X (time bins)XXT, N × Nneurons, RNthe row average
You have a trial-averaged table of N neurons by T time bins. Which statement is right?

Chapter 4

Two Ways to Say PCA

Keep as much of the data as you can, or lose as little as you can, and see why these are the same demand

You recorded 100 neurons and you must squeeze each time bin down to a single number. You pick one direction in neuron space and write down, for every bin, how far its counts reach along that direction. You throw the rest away. Which direction should you pick?

There are two sensible answers. First: pick the direction where the squeezed numbers are as big as possible, so you keep the most of the data. Second: pick the direction where what you throw away is as small as possible, so you lose the least. The lecture slides call these the two equivalent formulations of PCA, and the full derivation is in the course notes on PCA. This chapter shows why the two never disagree.

Formulation one: keep the most

Gather k directions as the columns of a table B, of shape d × k. Each column has length one, and the columns are at right angles to each other. Mathematicians write that as BTB = I, where I is the table with ones on the diagonal and zeros elsewhere. A set of directions like this is called an orthonormal basis.

Multiply the data by B. The product XB has shape (N × d) times (d × k), which is N × k: for each bin, its k squeezed numbers. The size of the data kept is the sum of all squared entries of XB. That sum has a name, the squared Frobenius norm, written ||XB||F2.

B̂pca
The answer: the best table of k directions. Its columns are the first k PCs.
arg maxB
“The B that makes the following as large as possible.” It returns the table, not the size.
||XB||F2
The sum of squares of all N × k squeezed numbers. The slide says: find the subspace that preserves the maximal sum of squares.
BTB = I
The rule that the columns of B are unit length and at right angles. Without it you could win by making B huge.

Formulation two: lose the least

Now undo the squeeze. Multiply XB by BT, of shape k × d. The product XBBT has shape N × d again: each bin’s counts, rebuilt from only its k squeezed numbers. The slide calls it the reconstruction of X in the subspace spanned by B. Each row of the reconstruction is the point’s shadow on the subspace. What is lost is X minus its shadow.

B̂pca
The same answer as before. That is the claim.
||X − XBBT||F2
The sum of squares of what the reconstruction misses. The slide says: minimise the sum of squares of the orthogonal component, the part of each point at a right angle to the subspace.

Why they agree: Pythagoras

Take one bin x and one direction u. The point splits into two pieces at a right angle: its shadow on the line, and the leftover that sticks out sideways. The pieces form a right triangle, and the triangle rule says the squared hypotenuse equals the sum of the two squared sides. So for every single bin, the point’s squared length equals the squared length of its shadow plus the squared length of its leftover.

Add that up over all bins. The data’s total sum of squares, ||X||F2, splits into kept plus lost. The total does not depend on the direction you pick, because it is fixed by the data. So the direction that makes kept as big as possible makes lost as small as possible. The two formulations name one winner.

A four-point example, with made-up counts already centred: (3, 1), (−3, −1), (1, 2), (−1, −2). The total is 9 + 1 + 9 + 1 + 1 + 4 + 1 + 4 = 30. Along the neuron 1 axis the squeezed numbers are 3, −3, 1, −1, so you keep 20 and lose 10. Along the neuron 2 axis you keep 10 and lose 20. Along PC 1, at 31.7 degrees, you keep 26.18 and lose 3.82. Every pair adds to 30.

For one direction, Chapter 1 already proved the winner. A unit direction z = a u1 + b u2 keeps zTCz = a2 s1 + b2 s2, with a2 + b2 = 1. That is a blend of s1 and s2. It is biggest when all the weight sits on the bigger number: a = 1, so z = u1. The first PC is the eigenvector of C with the largest eigenvalue, which is what the slide’s derivation reaches.

Turn the dial below. The yellow stubs are what you lose, and the warm feet on the axis are what you keep.

Keep the most, lose the least

Rotate the axis. Each dot drops a stub to the axis. The bar shows the kept share in warm and the lost share in yellow; the curve shows the kept sum of squares at every angle.

100°

Toy data: 40 centred made-up points, spreads 2.4 and 1.1, tilted 35 degrees, seeded for every visit. Kept is the sum over points of the squared reach along the axis, lost is the sum of the squared leftovers, and both are computed live. Their sum is the data’s total sum of squares and never moves.

Only one angle makes the bar nearly all warm, and it is exactly the peak of the curve. Slide the dial to the opposite quarter turn and the bar turns mostly yellow. The total never changes.

A direction keeps 70% of a cloud’s sum of squares. How much does the leftover hold, and why?

Chapter 5

Beyond PCA

Meet the cousins that fix what PCA cannot do, and learn what each one costs

Two electrodes sit in a patch of cortex. Each one hears a mixture of two neurons, a loud one nearby and a quiet one further away, in a different ratio on each wire. You want to unmix them: to recover what each neuron did on its own. Can PCA do that?

It cannot, and the reason teaches you what PCA really knows. PCA uses only the average and the covariance of the data. Those are the first and second moments (a moment is an average of a power of the data: the first power gives the mean, the second gives the spread). A round blob and a square with the same covariance look identical to PCA. This chapter follows the slides through five methods that look past it. The slide notes that PCA itself dates to 1901, by Karl Pearson.

ICA: use higher moments to find the axes

Independent components analysis (ICA) unmixes sources that are independent of each other and not bell-shaped. The slide credits Comon (1994), Bell and Sejnowski (1995), and Hyvärinen and Oja (2000). It makes three points. ICA uses higher-order moments to find its axes. Its axes need not be orthogonal. And it returns as many components as there are dimensions, so it demixes rather than reduces.

Here is the trick in one picture. Mix two independent sources that are uniform, so their joint cloud is a square. The mixing turns the square into a slanted parallelogram. PCA fits an ellipse to the parallelogram and returns two perpendicular axes. The sources, though, lie along the parallelogram’s two edges, which are not perpendicular. To find the edges, ICA first squashes the cloud so the covariance is round (this is called whitening). Then only a rotation is left to choose, and it picks the rotation that makes each projection as far from bell-shaped as possible.

A bell shape has an excess kurtosis of 0. Kurtosis is the average of the fourth power of the centred data, divided by the square of the variance, minus 3. A uniform source scores −1.2. When you add independent sources together, the sum gets closer to a bell, so a projection that mixes sources scores nearer to 0. The rotation that gives the most extreme scores unmixes them.

Find the edges of the parallelogram

Two electrodes each hear a mix of two independent neurons. Teal lines are your trial axes, warm dashes are PCA’s. Turn the rotation and watch the kurtosis, or press Find ICA axes.

10°

Toy data: 400 samples of two independent uniform sources, mixed by a made-up 2 by 2 table whose columns are 42 degrees apart, then centred. Whitening and kurtosis are computed live in your browser. The curve plots the sum of the two squared excess kurtoses, and the search tries every whole degree. Real ICA algorithms use faster optimisers.

At the best rotation both projections read about −1.2 and the teal axes lie along the parallelogram’s edges, 42 degrees apart. PCA’s axes stay at exactly 90 degrees however you turn the dial.

NMF and TCA: more shapes of structure

Non-negative matrix factorization (NMF; Paatero and Tapper 1994, Lee and Seung 1999) writes the data table as the product of two small tables whose entries must all be zero or positive. Spike counts are never negative, so this fits. The slide lists the gains: non-orthogonal axes, and results that are arguably easier to read, since the data are explained by a sum of positive features. A neuron’s response is built by adding parts, never by subtracting.

Tensor decomposition (the slide calls its neural version TCA, tensor components analysis; the credits are Hitchcock 1927, Tucker 1966 and Williams et al. 2018) starts from one more observation. Some datasets are naturally a block of numbers with three sides, here neurons by time by trial, and not a flat table. Flattening the block into a table throws away which numbers belong to which trial. A tensor method keeps all three sides: each component has a neuron weight, a time profile and a trial weight. The slide adds that the components are not orthogonal, and that recovering them is guaranteed in cases where PCA offers no such guarantee.

Kernel PCA, factor analysis and PPCA

Kernel PCA (Mika et al. 1998) first bends the data with a nonlinear map, then runs ordinary PCA on the bent data. A ring of points has no useful straight axis, but after the right bend it does. The slide adds two facts: the bent space can be infinite-dimensional, and the performance depends on the choice of kernel, the function that defines the bend.

Factor analysis (FA; Spearman 1904) and probabilistic PCA (PPCA; Tipping and Bishop 1999) turn the idea into a full probability model of the data: a few hidden factors drive the neurons, and each neuron adds its own noise. The slide stresses that this is a complete model of the dataset, as opposed to a plain dimensionality reduction. It is fitted by an iterative method, the EM algorithm (our EM for mixtures lesson shows the loop). Later lessons in the series, such as latent variable models, build on it.

MethodWhat it adds beyond PCAWhat it costs
ICAHigher-order moments; axes need not be orthogonal; demixingNumerical optimisation; as many components as dimensions
NMFNon-negative parts; non-orthogonal axesNumerical optimisation; components change when you add more
TCATensor data kept whole; recovery guaranteesNumerical optimisation; components change when you add more
Kernel PCANonlinear structureDepends on the kernel you choose
FA and PPCAA full probability model with noiseIterative fitting (EM)

The slide gathers the drawbacks of ICA, NMF and TCA in two lines. They need numerical optimisation, not merely a call to the SVD. And they are non-nested: ask for 5 components instead of 4 and all the components change. PCA is nested. Its first four PCs are the same whether you keep 4 or 40. The summary slide closes the PCA half with its checklist: PCA is an ellipse fit, centred or not, plotted after projection, row or column, and one SVD in three costumes. You have now met every item.

Why can ICA find the edges of a mixed-square cloud when PCA returns perpendicular axes?

Chapter 6

Linearity

Learn the one rule that makes a system linear, see that every such system is a matrix, and spot the common function that breaks it

A neuron in the auditory pathway listens to two tones at once. You play tone 1 alone and the neuron’s response grows by a certain amount. You play tone 2 alone and it grows by another amount. Now you play both together. Does the response equal the first amount plus the second? If it does, and if doubling a tone also doubles the response, you can predict the neuron’s answer to every mix of tones from just two measurements. That is the power of a linear system.

Before the lecture derives least squares on the board, the slides pause for this idea. The “but first” slide says that least squares assumes the data come from something linear, so we must be exact about what the word means.

The two properties

A linear system is a mapping f that takes an input vector and returns an output vector. Slide 24 gives it two properties. The first is homogeneity, also called scalar multiplication: f(a x) = a f(x), for any number a. Turn the input up by a factor of three and the output turns up by three. The second is additivity: f(x1 + x2) = f(x1) + f(x2). The response to two inputs together is the sum of the responses to each.

Slide 25 folds them into one equivalent rule, linear superposition.

f(a x1 + b x2)
Mix the inputs first, with weights a and b, then apply the system.
a f(x1) + b f(x2)
Apply the system to each input alone, then mix the outputs with the same weights. For a linear system the two orders give the same vector, for any scalars a and b.

Setting b = 0 gives homogeneity. Setting a = b = 1 gives additivity. Conversely, if both of those hold, then you can pull the weights out one piece at a time and reach the combined rule. So the two forms say the same thing.

Every linear system is a matrix

Slide 26 states the surprise: any linear function can be written as a matrix multiplication, f(x) = Ax, for some table A. Here is why, in three steps. Take an input with two entries, x = (x1, x2). Write it as x1 times the unit vector e1 = (1, 0) plus x2 times e2 = (0, 1). Then superposition says f(x) = x1 f(e1) + x2 f(e2). That is a weighted sum of two fixed output vectors, which is exactly what a matrix does. The columns of A are f(e1) and f(e2). Two measurements fix the whole system.

A worked check with a made-up neuron pair. Let A = [[2, 1], [0, 3]], so f(e1) = (2, 0) and f(e2) = (1, 3). Then f((1, 1)) = A (1, 1) = (3, 3), and the sum f(e1) + f(e2) = (2, 0) + (1, 3) = (3, 3). They match. With a = 2 and b = −1: f(2e1 − e2) = A (2, −1) = (3, −3), and 2 (2, 0) − (1, 3) = (3, −3). They match again.

The test question: is ax + b linear?

Slide 27 asks: is f(x) = ax + b a linear function? Slide 28 answers: no, it is called an affine function. Check it with the rule. Let a = 2 and b = 1, so f(x) = 2x + 1. Then f(1) = 3 and f(1 + 1) = f(2) = 5, but f(1) + f(1) = 6. Additivity fails by exactly b, the constant you added.

Neurons make this trap easy to fall into. A neuron almost always fires a little with no stimulus at all: its spontaneous rate, say 3 spikes per second. A model that says rate = (gain times stimulus) + 3 is affine. Its response to nothing is not zero, so doubling the stimulus does not double the rate. The line has a different name, but it still plays well with least squares, as Chapter 7 shows.

The device shows the two orders of mixing side by side. Choose a linear map, then an affine one, and change a and b.

Mix first, or mix last

Two input patterns go through a system. The blue arrows are the outputs for each alone. The teal dot mixes the inputs first. The warm ring mixes the outputs last. A yellow bar shows the gap, if any.

1.0
1.0
1.5

Toy system: A = [[1.0, 0.5], [−0.3, 1.2]], inputs x1 = (1.5, 0.5) and x2 = (−0.5, 1.5), and in the affine case a baseline added to both output channels (made-up spontaneous rates, in spikes per second). Every number is computed live.

In linear mode the dot and the ring always sit on top of each other. In affine mode they split, by the baseline times (1 − a − b). The gap is zero whenever the weights add up to one, which is why affine functions pass the superposition test only for such mixes.

A neuron fires 3 spikes per second with no stimulus and 7 per second at contrast 1. A model says rate = 4 × contrast + 3. Is it a linear function of contrast?

Chapter 7

Least Squares: The Question

Turn a pile of stimulus and spike-count pairs into one table, one error, and a bowl with a single lowest point

Back in the recording room. After each trial you have written one line: the contrast of the pattern, and the neuron’s spike count. Three trials are enough to start. Contrast 1 gave 2 spikes, contrast 2 gave 3 spikes, contrast 3 gave 7 spikes. These numbers are made up for this lesson. You want a straight line that predicts spikes from contrast.

The slides hand this part to the board. The divider reads “Least Squares (on board)”, so the written lecture notes carry the details, and we follow them. The notes open with a general setting. Someone gives you a stack of N inputs, x1 to xN, each a vector of d numbers. Each input has one scalar output yi. The notes call the input the regressor or predictor, and the output the dependent variable. The data arrive as pairs (xi, yi).

The model and its error

You want a weight vector w, a list of d numbers, such that yi is close to the dot product wTxi for every trial. The dot product multiplies each input entry by its weight and adds: a weighted sum. For one trial the residual is the miss, yi minus the prediction. Squaring it keeps misses positive, so over- and under-shoots cannot cancel, and it punishes big misses more than small ones. (Those reasons are ours. The notes simply choose the squared error.) Add the squares over all trials:

squared error
The one number we want as small as possible. It depends on w: change the weights and the error changes.
∑i=1N
Add up the term for every trial, from the first to the N-th.
(yi − xi · w)2
Trial i’s residual, squared. In the hero this was the yellow square on trial i.

Stacking trials into a table

A sum over trials is slow to push around on paper. The notes stack everything. Let Y be the column of outputs, of shape N × 1. Let X be the table whose rows are the inputs, of shape N × d. The notes call X the design matrix. Then the predictions for all trials at once are Xw, shape (N × d) times (d × 1), which is N × 1. The residuals are Y − Xw. The sum of their squares is the squared length of that vector, written ||Y − Xw||2.

The notes then state the answer without proof, and put a hat on w to say it is an estimate from data:

ŵ = (XTX)−1 (XTY)   shapes: (d × d) times (d × 1) = d × 1

Chapters 8 and 9 prove it two ways. First let us use it, so the proofs have a target.

A full worked example

Real neurons fire at zero contrast, so allow a baseline. Give each trial two inputs: its contrast and a constant 1. The weights are then (gain, baseline), and this is exactly the “column of ones” fix for the affine problem of Chapter 6. The design matrix has rows (1, 1), (2, 1), (3, 1), and Y = (2, 3, 7).

  1. Form XTX. The entries are sums of products. Top left: 1 + 4 + 9 = 14. Off the diagonal: 1 + 2 + 3 = 6. Bottom right: 1 + 1 + 1 = 3. So XTX = [[14, 6], [6, 3]].
  2. Form XTY. Top: 1·2 + 2·3 + 3·7 = 29. Bottom: 2 + 3 + 7 = 12. So XTY = (29, 12).
  3. Invert. The determinant is 14·3 − 6·6 = 6. For a 2 × 2 table, swap the diagonal, negate the off-diagonal and divide by the determinant: the inverse is (1/6) [[3, −6], [−6, 14]].
  4. Multiply. ŵ = (1/6) (3·29 − 6·12, −6·29 + 14·12) = (1/6) (15, −6) = (2.5, −1).
  5. Check. Predictions: 1.5, 4, 6.5. Residuals: 0.5, −1, 0.5. Squared error: 0.25 + 1 + 0.25 = 1.5.

The best line is spikes = 2.5 × contrast − 1. Its negative baseline is a warning: counts cannot be below zero, so the straight line is only an approximation, and it works best inside the range of contrasts you measured. Now play with it below. Two sliders set your own line, and the picture on the right shows the error of every possible line at once.

Walk down the error bowl

Each line has two weights, a slope and an intercept. The right-hand map colours every pair of weights by its squared error: warm is high, dark is low. Your pair is the teal dot.

1.00
1.50

The three trials are the worked example: contrast 1, 2, 3 and spike counts 2, 3, 7, made up for this lesson. The map covers slopes 0 to 5 and intercepts −4 to 3 and is computed live, with colour on a logarithmic scale of the squared error. The green marker is the formula’s answer, (2.5, −1).

Wherever you stand, the map slopes down toward one point. There is a single bottom and no side pits where you could get stuck. That is the property that lets a single formula, and not a search, find the answer. Chapter 9 shows why the bowl has this shape.

You have 100 trials, 5 neurons’ spike counts as inputs, and a constant of 1 as a sixth input. What is the shape of the design matrix X, and of the weight vector?

Chapter 8

Derivation One: The Right Angle

Prove the least squares formula by dropping a perpendicular from the data onto the space your inputs can reach

Look at the three trials of Chapter 7 in a new way. Until now each trial was a dot on a plot. Now take the three spike counts together, (2, 3, 7), and treat them as one point in a three-dimensional space where each axis stands for a trial. The contrast column (1, 2, 3) is another point in that space. So is the column of ones, (1, 1, 1). The course notes make this move on purpose, and it turns an algebra problem into a picture. The notes say they give two derivations, and that the first, the one covered in class, is easier and shows the geometry.

The space your inputs can reach

Write X by its columns instead of its rows: X = [X1 … Xd]. Each column is a vector with N entries, one per trial. Now think about all the predictions you could ever make. A prediction is Xw = w1X1 + … + wdXd: a mix of the columns. The set of all such mixes is the subspace spanned by the columns, a flat sheet through the origin. With two columns in three dimensions it is a plane, like a sheet of glass. The notes say it is a d-dimensional subspace inside the larger N-dimensional space that holds Y.

In general Y does not lie on the sheet. If it did, the model would predict every trial perfectly. So least squares asks: which point of the sheet is closest to Y? The squared distance from Y to the point Xw is exactly the squared error, ||Y − Xw||2. Minimising the error means finding the closest point.

The closest point makes a right angle

Anyone who has dropped a plumb line knows the answer: the shortest path from a point down to a sheet goes straight down, at a right angle to the sheet. The notes say this: the optimal combination corresponds to dropping a line from Y to the subspace at a right angle. The error vector, Y − Xw, is the residual vector, and it must be orthogonal (at a right angle) to every column of X. Two vectors are orthogonal when their dot product is zero.

(Y − Xw)
The residual vector: what the model misses, one entry per trial.
Xj
Column j of the design matrix, for example all the contrasts, or all the ones.
0
A zero dot product means a right angle. The residual leans toward none of the columns, so no tweak of the weights can shrink it.

Why a right angle is enough

Suppose the residual r = Y − Xw is orthogonal to every column. Pick any other weights v. The error at v is ||Y − Xv||2. Write Y − Xv = (Y − Xw) + X(w − v). The first piece is r. The second piece, X(w − v), is a mix of the columns, so it lies in the sheet, and r is at a right angle to it. Pythagoras applies: the squared length of the sum is the sum of the squared lengths. So the error at v equals ||r||2 plus ||X(w − v)||2, which is at least ||r||2. No other weights do better. The right angle is not only necessary but sufficient.

From the right angle to the formula

Collect the d conditions into one matrix equation, as the notes do: (Y − Xw)TX = 0T, a row of d zeros. Take the transpose of both sides, and multiply out:

  1. Start. XT(Y − Xw) = 0.
  2. Expand. XTY − XTXw = 0.
  3. Rearrange. (XTX) w = XTY. These are called the normal equations, because “normal” is another word for perpendicular.
  4. Solve. Multiply both sides by (XTX)−1: w = (XTX)−1XTY. The notes end this derivation with “Hooray!”

Check it on the Chapter 7 numbers. The columns are X1 = (1, 2, 3) and X2 = (1, 1, 1). The solution is w = (2.5, −1), so the prediction is Xw = (1.5, 4, 6.5) and the residual is r = (0.5, −1, 0.5). Dot r with X1: 0.5 − 2 + 1.5 = 0. Dot r with X2: 0.5 − 1 + 0.5 = 0. Both zero. And Pythagoras holds: ||Y||2 = 4 + 9 + 49 = 62, ||Xw||2 = 2.25 + 16 + 42.25 = 60.5, ||r||2 = 1.5, and 60.5 + 1.5 = 62.

The picture needs only two dimensions to show. Let the data space have two trials, and let X have one column, X1 = (2, 1). The sheet is just a line. Move the weight, or drag Y if you have a mouse, and watch the residual.

Drop the perpendicular

The purple line is every prediction Xw. The blue dot is Y, your two spike counts. The yellow segment is the residual. Change w, or pick another Y, and find where the segment meets the line at a right angle.

0.40

Toy setting: two trials, one input column X1 = (2, 1), so the sheet is a line. The best weight is the dot product of X1 and Y divided by the dot product of X1 with itself, the one-column case of the formula. With a mouse you can also drag the blue point; on a touch screen use the buttons. All numbers are computed live.

For Y = (1, 3) the best weight is (2 + 3) / 5 = 1.00. The prediction is (2, 1), the residual is (−1, 2), and the dot product of the residual with X1 is −2 + 2 = 0. At any other weight the residual leans one way or the other, the dot product is not zero, and the squared length is bigger.

At the least squares solution, what is true of the residual vector Y − Xw?

Chapter 9

Derivation Two: Calculus, and Where It Breaks

Reach the same formula by setting a slope to zero, then find out what goes wrong when two of your neurons say the same thing

You already know the answer from Chapter 8. A second proof is worth having for two reasons. It uses a tool you will meet again in every later lecture: differentiate the error and set the slope to zero. And it shows exactly which assumption the formula needs, which is where the trouble starts.

Two rules for slopes of vectors

In one variable you know that the slope of a x is a, and the slope of a x2 is 2 a x. The notes give the same two rules for vectors. First they define the gradient of a function f of a vector x: it is the vector of slopes, one for each entry of x, written ∇f or ∂f/∂x. It points uphill.

∂(aTx)/∂x = a
Rule 1, the slope of a linear function: just like the slope of a x being a. Here aTx is a weighted sum of the entries of x.
∂(xTAx)/∂x = 2Ax
Rule 2, the slope of a quadratic function, for a symmetric table A (one that equals its own transpose): just like the slope of a x2 being 2 a x. For a table that is not symmetric the rule is (A + AT)x, which equals 2Ax when A is symmetric.

The derivation, line by line

Start with the squared error, SE = (Y − Xw)T(Y − Xw). Multiply it out. Two cross terms appear, YTXw and wTXTY. Each is a single number, and a single number equals its own transpose, so the two are equal and combine into one.

  1. Expand. SE = YTY − 2 wTXTY + wTXTXw.
  2. Differentiate in w, term by term. YTY does not contain w, so its slope is 0. The middle term is linear in w with a = XTY, so Rule 1 gives −2XTY. The last term is quadratic with A = XTX, which is symmetric, so Rule 2 gives 2XTXw.
  3. Set the slope to zero. −2XTY + 2XTXw = 0.
  4. Rearrange. XTXw = XTY. The same normal equations as Chapter 8.
  5. Solve. w = (XTX)−1XTY. The notes comment: “Easy, right?”

The notes say that a zero slope implies a local optimum. We can say more. The slope of the slope is 2XTX, and wTXTXw = ||Xw||2 can never be negative. So the error curves upward in every direction. It is a bowl, as the map of Chapter 7 showed, and its single flat spot is the single bottom.

The assumption inside the proof

The last step needs an inverse. The notes add the condition: XTX must be full rank, meaning the inverse exists. They say this implies that N is larger than d and that the rows of X are not all linearly dependent. In plain words: you need more trials than weights, and the inputs must not repeat each other.

What if they nearly repeat each other? This is the usual case in neural data. Two neighbouring neurons on a probe often fire together. Use both as inputs to predict a third neuron, or a hand velocity, and the formula still runs, but the answer turns shaky.

The failure: symptom, measurement, fix

Symptom. You fit the same model on two recordings of the same experiment. The predictions are equally good both times. But the weights on the two neighbouring neurons are +14 and −12 in the first fit, and −3 and +5 in the second. The weights swing wildly and cancel each other.

Measurement. Let the two inputs have correlation ρ. Then XTX is about N times [[1, ρ], [ρ, 1]], whose two eigenvalues are N(1 + ρ) and N(1 − ρ). Their ratio is the condition number, (1 + ρ)/(1 − ρ): 3 at ρ = 0.5, 19 at 0.9, 199 at 0.99. A big condition number means the bowl of Chapter 7 is a long, flat canyon. Many weight pairs sit near the bottom. Noise slides the answer along the canyon floor. The spread of each weight grows by a factor 1/√(1 − ρ2), which is 7.1 at ρ = 0.99. The sum w1 + w2 stays steady, because the data really do pin down that sum.

Two neurons that say the same thing

Two input neurons both drive an output with true weights (1, 1). Each blue dot is the fitted weight pair from one noisy repeat of the experiment. Raise the correlation and watch the cloud stretch along the green diagonal.

0.20

Toy experiment: 30 trials, two inputs with sample correlation exactly the value shown, an output equal to the sum of both inputs plus unit-size noise, and 40 repeats that share the same inputs but have fresh noise. Each repeat is solved with the normal equations. Dots beyond the plot edge are pinned to it. Seeded, so the first picture is the same on every visit.

Fix. There are three, from cheapest to richest. Drop or merge one of the twin inputs, or run PCA first, as in Chapter 1, and regress on the top PCs instead, which have no correlation by construction. Add a small penalty that keeps the weights small, for example (XTX + λI)w = XTY, called ridge regression, which later lessons cover. And never compute the inverse explicitly: ask a library routine that solves the system by the SVD (Chapter 10 shows the call), because it stays accurate when the canyon is narrow.

Two neighbouring neurons fire almost identically. You use both to predict a third. What do you expect from least squares?

Chapter 10

The Field Guide

Keep the numbers, the terms, the checklist and the code, and see where to go next

The lecture finished PCA and then built least squares. PCA fits an ellipse to centred data. Its PCs are the ellipse’s axes, and its singular values say how much of the data each axis holds. Row PCA and column PCA are two sides of one SVD, but they centre differently. Linear means superposition, and every linear map is a matrix. Least squares drops a perpendicular from Y onto the space the inputs reach, and one formula gives the weights, by a right angle or by calculus. It breaks when the inputs nearly repeat each other.

The numbers worth remembering

ItemValueWhere
Warmup: z = a u1 + b u2zTCz = a2 s1 + b2 s2; with s = 9, 1 and a = 2, b = 1 it is 37Ch 1
Raw versus centred PCARaw eigenvalues 4085.3 and 62.7, PC 1 at 27.2° (the centre sits at 27.6°); centred eigenvalues 80 and 30, PC 1 at −26.6°Ch 2, six made-up bins
Row and column PCASame singular value 5 (variance 25) raw; 10 after centring each neuron; 2.5 after centring each time binCh 3, a 2 by 3 table
Kept plus lostAlways the total: 26.18 + 3.82 = 30 along PC 1; 20 + 10 = 30 along neuron 1Ch 4, four points
Excess kurtosis0 for a bell; −1.2 for a uniform sourceCh 5
Affine, not linearf(x) = 2x + 1 gives f(2) = 5, but f(1) + f(1) = 6Ch 6
The worked fitXTX = [[14, 6], [6, 3]], XTY = (29, 12), w = (2.5, −1), squared error 1.5Ch 7
The right angler = (0.5, −1, 0.5) has zero dot product with (1, 2, 3) and (1, 1, 1); 60.5 + 1.5 = 62Ch 8
Condition number(1 + ρ)/(1 − ρ): 3 at 0.5, 19 at 0.9, 199 at 0.99; weight spread grows by 7.1 at 0.99Ch 9

The terms in one line each

TermIn plain words
Second moment matrix CXTX: sums of products of pairs of neurons’ counts.
Principal componentA unit direction in neuron space; a major axis of the fitted ellipse.
CenteringSubtract the average first, so C becomes a covariance.
Row PCA, time PCsRows are the points; the PCs are right singular vectors, curves over time.
Column PCA, neuron PCsColumns are the points; the PCs are left singular vectors, weights over neurons.
Frobenius normSquare every entry of a table, add them, and take the root.
Linear superpositionf(a x1 + b x2) = a f(x1) + b f(x2).
Affine functionA linear map plus a constant; it fails superposition by that constant.
Design matrix XOne row per trial, one column per input, shape N × d.
ResidualY − Xw: the miss on every trial.
Normal equations(XTX) w = XTY: the residual is perpendicular to every column.
Condition numberBiggest over smallest eigenvalue of XTX; large means unstable weights.

The checklist

  1. Centre first. Say which way: subtract each neuron’s average (neuron PCs) or each time bin’s average (time PCs).
  2. Look at the projection. A good variance share does not prove the cloud is an ellipse.
  3. Check the shape of X. N rows for trials, d columns for inputs, and N larger than d.
  4. Add a column of ones if the neuron has a baseline.
  5. Test the right angle. XT(Y − Xw) should be zero.
  6. Check the condition number of XTX before you read any single weight.

The core as code

pythonimport numpy as np

# ---- PCA of N time bins x d neurons (one row per bin) ----
Xc = X - X.mean(axis=0)                          # centre each neuron: (N, d)
U, s, Vt = np.linalg.svd(Xc, full_matrices=False)  # Xc = U diag(s) Vt
pcs  = Vt[:k].T                                     # (d, k): the first k PCs as columns
proj = Xc @ pcs                                  # (N, k): the flat view of the cloud
var  = s**2 / len(Xc)                             # variance along each PC (divide by N)
# np.cov(X, rowvar=False) gives the same matrix, but divides by N - 1

# ---- trial-averaged table R: N neurons x T time bins ----
Rn = R - R.mean(axis=1, keepdims=True)             # centre each neuron over time
neuron_pcs, sn, _ = np.linalg.svd(Rn, full_matrices=False)   # (N, N) left singular vectors
Rt = R - R.mean(axis=0, keepdims=True)             # centre each time bin over neurons
_, st, time_pcs = np.linalg.svd(Rt, full_matrices=False)      # rows are the time PCs, (N, T)

# ---- least squares: spikes from contrast, with a baseline ----
X = np.column_stack([contrast, np.ones(len(contrast))])  # (N, 2) design matrix
w = np.linalg.solve(X.T @ X, X.T @ y)       # normal equations: (2,)
w, *_ = np.linalg.lstsq(X, y, rcond=None)       # safer: an SVD-based solver
r = y - X @ w                                     # residuals, (N,)
assert np.allclose(X.T @ r, 0)                    # the right angle: r is orthogonal to every column
cond = np.linalg.cond(X.T @ X)                # large means the weights are unstable

On the worked example, contrast = [1, 2, 3] and y = [2, 3, 7], this returns w = [2.5, −1], r = [0.5, −1, 0.5] and a condition number of 46.1.

Where it sits in the field

PCA is the oldest tool for summarising a population, and the ellipse view carries into the later lectures. Least squares is the base of linear decoding in brain-computer interfaces, because predicting hand velocity from spike counts is exactly this fit. The next steps in the series add noise models that suit spike counts, and hidden-state models that smooth over time.

Keep going

The takeaway. PCA and least squares both ask a question of squares and right angles. PCA asks which direction keeps the most. Least squares asks which weights leave a residual at a right angle to every input. Centre your data, look at your projection, test the right angle, and distrust any weight whose input has a twin.

Now press Present or Teach and explain, from memory, why the first PC of uncentred data points at the cloud, why row and column PCA differ once you centre, and why the residual of a least squares fit is at a right angle to every input. Then go back to the line at the top, drag the slope away from its best value, and say what the yellow squares add up to.

NEU 560 Lecture 4: PCA part II and Least Squares
NEU 560 course site · Lecture 3, PCA · Back to Gleams