Stanford CS221 | Autumn 2025 | Lecture 14: Bayesian Networks and Learning — Annotated Lecture Notes & Key Takeaways

Course Information

Institution: Stanford

Original Course: Stanford CS221 | Autumn 2025 | Lecture 14: Bayesian Networks and Learning

Instructor Bio: This lecture is delivered by Percy Liang, Associate Professor of Computer Science at Stanford University and core faculty of the Stanford Institute for Human-Centered Artificial Intelligence (HAI). Percy Liang leads the Stanford Natural Language Processing Group and the Center for Research on Foundation Models (CRFM). His research spans the theoretical foundations and practical systems of artificial intelligence, including machine learning, natural language processing, AI alignment, and rigorous model evaluation. He received his PhD in Computer Science from the University of California, Berkeley and his BA in Mathematics from Harvard University. His work has been recognized with the NSF CAREER Award, Google Faculty Research Award, and multiple best paper awards at top-tier AI conferences. He has taught CS 221 at Stanford for over a decade, shaping foundational AI education for thousands of students.

Course Description: This lecture addresses the problem of learning Bayesian network structures and parameters from observational data. It covers parameter learning for known network structures using maximum likelihood estimation and Bayesian estimation, then introduces structure learning algorithms for discovering causal graphical models from data. The lecture also discusses the fundamental distinction between correlation and causation, and the core challenges of causal inference from observational data.

Lecture 14 is the point where Bayesian networks stop being something you're handed and become something you have to build. The title promises learning, and the lecturer says in the first minute that today is about the parameters — but about thirteen of the sixty minutes go to review first. That's worth knowing up front if you're deciding whether to watch the whole recording or jump ahead: the new material starts around (13:00), when he asks out loud where all these probabilities were supposed to have come from.
 

The Re-run: Network Structure, Inference, and Why the Tables Are Shaped That Way

 

The review is genuinely quick. A Bayesian network is framed as a compressed way of writing down a joint distribution over a bunch of variables — he uses the image of a database holding facts about the world. You pick your variables, draw a directed acyclic graph connecting the ones you think influence each other, and then attach one local conditional distribution to every node. He stresses a detail that trips people up: the tables go with nodes, not edges.
 

Each table is essentially a small tensor holding a probability for every value the node can take, and when the node has parents, you get one such distribution for every complete assignment of those parents. He spends a moment on the normalization asymmetry, and it's a useful moment: within a conditional table, the numbers only sum to one if you fix the parents and sum over the child; in the full joint table, everything sums to one. Obvious once written down, easy to muddle when you're staring at a board full of fractions.
 

Inference gets compressed to about ninety seconds. Exact inference means filtering the joint table down to rows consistent with the evidence, collapsing away the variables you don't care about, and dividing through by the probability of the evidence. Rejection sampling means writing a probabilistic program, generating whole assignments, discarding the ones that contradict what you observed, and counting. Gibbs sampling means starting from a full assignment and repeatedly resampling one variable at a time given all the others, with evidence variables pinned in place. If you want more, he points at CS228.
 

My honest read: for anyone catching up from notes rather than from lectures 12 and 13, this stretch is too fast to be self-contained. He says outright that he isn't going to walk through it again.
 

Conditional Independence as a Game of Blocked Paths

 

This is the section he deliberately re-teaches. He says he's presenting it differently than last time, hoping it lands better, with more examples (5:27). The framing is path-based: two variables are independent given a third set when every path between them is blocked, and a path is blocked when it contains one of three shapes.
 

A chain, X → C → Y, is blocked if you condition on the middle node. A fork, where C points out to both X and Y, is also blocked when you condition on C. The V-structure is the one that runs backwards: where two arrows converge on a node Z, the path is shut by default and only opens up when you condition on Z or on something downstream of Z.
 

He then puts the two contrasting pairs on the board. With a V-structure and the child unobserved, the two parents are independent; observe the child and they become dependent — that's explaining away, the burglary-and-earthquake effect, where hearing the alarm suddenly makes two unrelated causes compete to explain it. With a fork, the pattern inverts: unobserved common cause means dependence, and conditioning on it buys you independence. His claim is that holding those two pairs in your head gets you most of the way through any independence question.
 

There's a small stumble at (7:36) where he corrects himself mid-sentence about whether the V-structure is blocked when you condition or when you don't. It gets sorted out, but it's a fair signal of where the difficulty actually lives — the rule that runs opposite to the other two is the one that gets misremembered.
 

Then he runs an interactive round, adding nodes to a graph and having the class vote by show of hands on whether A and B came out independent. Two durable takeaways survive into the transcript. First, the V-structure rule extends to any descendant of the converging node, not just the node itself. Second, unobserved leaves can effectively be deleted: if a branch ends in a node nobody conditions on, there's no path through it (11:00).
 

The payoff he draws is computational rather than philosophical (12:03). Independence means pieces don't interact, which means they can be handled separately. The example is a variable Z sitting above a fan of data points x₁ through xₙ — condition on Z and every x becomes independent of every other x, so a Gibbs sampler can update the whole batch in one shot instead of crawling through them one at a time.
 

One limitation worth flagging: the middle of that interactive round, roughly the ten minutes from (9:30) to (12:00), does not reconstruct from audio. It's full of "this" and "that" pointing at chalk. If you're studying from the recording, that's the stretch where you need the video and not the captions.
 

Learning the Tables in Bayesian Networks: Count, Normalize, Repeat

 

The pivot lands at (13:00). Everything so far assumed the probabilities appeared out of thin air, which he calls unprincipled. So: learn them from data. The setting for today is fully observed — every training example pins down a value for every single variable in the network — and the output of learning is always the same kind of object, the collection of numbers sitting in the local conditional tables.
 

What follows is a deliberately repetitive ladder of examples, and I think the repetition is the teaching strategy rather than filler.
 

Start with one node: a movie rating that takes values one through five. The network is a single node with no edges, and the parameters are five numbers. He notes in passing that a sharp student would say four, since they have to sum to one, then waves it off.
 

Add a second variable, genre, with two values, drama and comedy. Now there are two tables — one for genre, one for rating given genre — and estimation means counting how often each genre appears, counting how often each (genre, rating) pair appears, and turning each pile of counts into a distribution separately.
 

Go to three variables with a genuine V-structure: genre and an award indicator both pointing into the rating. He flags that you might expect the V-structure to cause trouble here, since it caused trouble for independence, then says that for learning it's surprisingly dull. You just count triples.
 

The principle that falls out is simple and worth writing on your own cheat sheet: to estimate one local conditional distribution, ignore every variable outside that node and its parents. And the parents operate as a single block. The number of distributions you end up normalizing equals the number of parent-value combinations, and you condition on all parents at once even though the edges in the drawing look like separate wires. A second example with the arrows reversed — one genre node feeding two users' ratings — makes the bookkeeping concrete, since the counts for the first user's table never touch the second user's value at all.
 

One Table Powering Many Nodes

 

The parameter-sharing section is the conceptual heart of the lecture, and it opens with a motivation that's easy to sympathize with (30:00). Picture a thousand users, each with a personal rating table. Someone who watched one movie gives you almost nothing to estimate from. So instead of a thousand tables, fit one.
 

The mental picture he offers is the most memorable thing in the hour: think of the conditional tables as objects floating outside the graph, and the graph as a wiring diagram that hooks those tables up to nodes to bring them to life. Under that picture, sharing just means one table is spliced into two places. The drawn network looks identical; the parameterization underneath is different, and that difference was invisible during inference.
 

A student asks — around (34:03) — why not simply merge the two user nodes into a single variable. The answer is clean: the data still contains two distinct ratings per movie, and a merged node has nowhere to record both. This is the moment where variables and parameters, which had been one-to-one all quarter, visibly come apart.
 

The sharpest line of the lecture follows shortly after (35:01). During inference you only ever read from the tables, so whether two nodes share one is irrelevant — they're just numbers. During learning you write to them, and then it matters enormously. The analogy he reaches for is pass-by-reference versus pass-by-value in programming: mutating a shared object in one place shows up somewhere else.
 

The decision of whether to share is left as a modeling judgment, with the trade-off stated plainly. Fewer parameters means you need less data per user; more parameters means more flexibility to capture users who genuinely differ. Lots of data → split freely. Thin data → be stingy.
 

The Hidden Markov Model, and the General Recipe

 

Hidden Markov models arrive as the big sharing payoff. There's a chain of hidden states and a parallel chain of observations, and the whole joint distribution is built from three parameter objects: a start distribution, a transition distribution, and an emission distribution. Same transition table reused at every step, same emission table reused at every step. He mentions these used to be common in speech recognition and computational biology, and can be used for object tracking, but doesn't go further into applications.
 

The observation that makes the point is about scale: three time steps, three tables. A hundred time steps, still three tables. He's also upfront that for today's estimation he assumes you can actually see the hidden states during training, which is a strong assumption he sets aside rather than defends.
 

Then the general algorithm. You define a set of labels — he calls it D — naming the types of local conditional distribution in the model; for an HMM that's start, transition, emission. Those labels are explicitly not variables. Every node records which label powers it plus the names of its parent variables, and sharing is just two nodes carrying the same label. Estimation becomes: walk the training data, and for each variable look up its label and its parents' values in the current example, then bump a counter keyed by (label, parent values, node value). Once the counting is done, normalize over node values within each (label, parent values) group. He claims roughly fifteen lines of code, and notes that the pseudo-counts appearing in his sketch should be ignored for now.
 

Two caveats he states himself: the graph structure is assumed known and fixed, with structure learning explicitly out of scope for the class; and "supervised learning" is a slightly misleading name for what's happening, since what really matters is full observability, with a partially observed version coming later.
 

Maximum Likelihood, and the Cliffhanger at the End

 

The last fifteen minutes answer the question of why counting is legitimate. It's not merely sensible — it is exactly maximum likelihood estimation. He connects it outward, saying that training a large language model by minimizing cross-entropy or perplexity is the same principle in a much bigger costume.
 

The objective is to pick parameters that make the observed data as probable as possible. Written out, that's a product of per-example probabilities; take logs, and it becomes a sum of log-probabilities, which is negative cross-entropy. Then comes the claim I found most striking: in the fully observed case, this optimization problem has a closed-form solution. No gradients, no step-size tuning, no convergence worries. You count and you normalize.
 

He gestures at the one-variable derivation — introduce a Lagrange multiplier for the constraint that probabilities sum to one, differentiate, set to zero, solve — and lands on the familiar count-over-total. The intuition he offers is that you push mass onto what you saw and leave zero on what you didn't, becauseProbability mass spent on unseen outcomes is wasted. He calls his own explanation sketchy, which is fair.
 

The two-variable argument is better, and it's the part I'd actually study. The likelihood product can be regrouped so that every factor touching the same table sits together — all the genre factors in one pile, all the drama-conditioned rating factors in another, all the comedy-conditioned ones in a third. Those piles share no parameters, so they're independent optimization problems, and each one is just the one-variable case again. That's the real proof that the mindless per-table counting is optimal, and it's genuinely satisfying.
 

Where I'd push back: the "you can now sleep at night in comfort" line slightly oversells what was demonstrated. The one-variable result is asserted rather than derived, and the multi-variable argument is shown on a two-variable example and generalized in a single sentence. If you want the actual Lagrange algebra, you'll be doing it yourself.
 

And then the lecture runs out. At (59:02) he announces Laplace smoothing, sets up the toy case of a single observed rating, and asks whether we genuinely believe the distribution that pure maximum likelihood would give us. The answer, of course, is no — one observation of a four would put all mass on four and zero on everything else. That's presumably where the pseudo-counts he told us to ignore come back in, and it's also where the practical nightmare case lives: a parent-value combination that never appears in training at all, leaving you normalizing a row of zeros.
 

The transcript ends mid-sentence at 59:57, so none of that gets developed. If you're working from the same recording I am, treat smoothing as assigned reading rather than covered material.
 

Taken as a whole, the lecture's real contribution isn't the counting — that part is trivial, and he jokes you could have guessed it without showing up. It's the shift in viewpoint: once you start learning, the tables in a Bayesian network become objects with identity and ownership, and deciding which nodes share which table turns out to be the interesting modeling decision. Inference never forced you to care. Learning does.
 

Content Disclaimer:

This article is for general reference only and does not constitute professional R&D guidance, production process advice or quality certification. All material performance data has specific test premises; readers should verify parameters against actual equipment and working conditions.