LLM Tuning — Annotated Lecture Notes & Key Takeaways

I walked into Lecture 5 without having seen Lecture 4, so the opening recap was genuinely useful rather than filler. The lecturer spends roughly ten minutes rebuilding the pipeline we're supposed to already know, and by the end of it the shape of LLM tuning is clear: two big stages everyone talks about, and a third stage that this whole lecture is about. If you missed it, the good news is that the third stage is motivated extremely carefully — the lecturer gives three separate reasons for why it exists before touching a single formula. The bad news is that the back half moves fast, and he says out loud at one point that he's behind schedule.

There's also some housekeeping at the very top. The midterm had just happened, the exam and solutions are posted, and the final will follow the same format but draw on lectures five through nine. Worth knowing if you're auditing.
 

Where LLM Tuning Sits: Pre-training, SFT, and a Third Step

The recap frames model building as a ladder. The bottom rung is pre-training: you take an initialized model and throw enormous amounts of text and code at it. It's slow, it's expensive, it's compute-heavy, and it's the part where all the parallelism machinery from last time lives — data parallelism, the ZeRO variants, a quick pass over model parallelism. What falls out at the end is a model that has absorbed the structure of language and code and can do exactly one thing: guess the next token. A superb autocomplete engine and a useless assistant, which is roughly how the lecturer puts it.

The second rung is supervised fine-tuning. Same next-token objective, completely different data: much smaller, far higher quality, aimed at a specific behavior. This is where a raw model becomes something that acts like a chat assistant. The point that stuck with me is that at this stage the model isn't learning new knowledge — it already knows what language and code are. It's learning how to behave. LoRA came up here as the parameter-efficient trick from last lecture: instead of moving all the weights, you inject low-rank matrices and tune only those.

Then the lecturer asks the question the rest of the session answers: what's left after SFT? His answer is that the model can do the right thing in the wrong way. It might answer correctly but in a tone nobody wants, or in a way that isn't safe. Tone, friendliness, safety — those are the things the third stage, preference tuning, is meant to fix.

He gives three reasons why you'd build a whole new stage instead of just patching the SFT dataset.

First, data is cheaper to produce. Writing a great poem from scratch is hard; being shown two poems and picking the better one is easy. Preference data asks for the easy thing.

Second, SFT datasets are fragile with respect to prompt mix. If one type of prompt is overrepresented, the model leans that way. So every time you want to correct one misbehavior by adding an SFT example, you have to think hard about whether that addition skews the distribution.

Third, SFT data has to be genuinely excellent, so sweeping up every misstep into it is a grind.

What I appreciated is that he immediately undercuts his own pitch. If your model is misbehaving badly and broadly, he says, that may be a symptom that your SFT dataset is broken, and preference tuning is not the universal remedy. That felt honest in a way lectures often aren't.

One more conceptual point arrives via an audience question: preference tuning lets you inject negative signal. SFT only ever says "produce this." It never says "don't produce that." A preference pair says both at once. The same question-answer also clarifies that LoRA and preference tuning aren't alternatives — LoRA is about how many parameters you touch, preference tuning is about what objective you optimize, and you can absolutely combine them.
 

Collecting Preference Data, and Why Pairwise Comparisons Won

Before any method gets introduced, the lecturer deals with the unglamorous question of where preference pairs come from. He lays out three formats. Pointwise: score each response in isolation, say 0.9 or 0.2. Pairwise: look at two and pick the winner. Listwise: rank a whole set of n responses. Pointwise is dismissed as punishing for human raters — nobody has a calibrated internal scale for "how good is this poem." Listwise is easier than pointwise but still fiddly. Pairwise wins, and the rest of the lecture assumes it.

Generating the two candidate responses is straightforward: run the same prompt through the model twice at a non-zero temperature and keep both samples. The prompts themselves are supposed to mirror what real users ask, so they typically come from logs or from a deliberately curated prompt set. Then someone or something has to judge. Options listed are human raters, LLM-as-a-judge (flagged as a topic for a later lecture), and rule-based metrics like BLEU and ROUGE, which he notes have fallen out of favor here.

He also mentions a finer-grained scale — much better, better, slightly better, and the mirror image on the downside — but then talks himself out of it. Subjectivity makes fine gradations unreliable, so most real datasets are binary: is A better than B, yes or no.

A second recipe gets a brief mention: go find a response in your logs that you didn't like and rewrite it by hand into the response you wanted. That's the more labor-intensive path, and he doesn't dwell on it.
 

RL in Twenty Minutes, Then the Two Halves of RLHF

This is the section where I'd have been lost without the hand-holding. The lecturer polls the room for RL experts, gets nobody, and starts from zero.

The vocabulary is standard: an agent sits in a state, takes an action according to a policy, and receives rewards from an environment. The mapping onto language models is the part worth writing down. The agent is the LLM itself. The state is the input accumulated so far. The action is the choice of the next token. The environment is effectively the vocabulary. And the policy is just the next-token probability distribution the model already emits on a forward pass — which means, and this is the part I underlined, the policy isn't a new object bolted on top of the model. It's the thing the model was already computing.

Then the rewards get bolted on, and they apply to the whole completion, not to individual tokens. Two audience questions probe this, and the answers are interesting. On cost: yes, it's expensive, but people batch, and it isn't intrinsically worse than other training procedures. On signal strength: RLHF is comparatively sparse, because SFT gives you a learning signal at every token while this gives you roughly one signal per finished response.

From there the lecturer splits RLHF into two stages. Stage one: learn to tell good completions from bad ones. Stage two: use that signal to push the model around. He also makes a naming point I hadn't thought about — the "human feedback" in RLHF refers specifically to the labels used to train the reward model. Train that model on AI-generated preferences instead and you're doing RLAIF, not RLHF.
 

Building the Reward Model from Bradley–Terry

Stage one gets the most careful treatment in the entire lecture, and it's the section I'd rewatch if I were you.

The goal is a function that takes a prompt and a response and returns a single number saying how good that pairing is. Crucially the prompt has to go in too, because the same response can be great for one question and terrible for another.

The machinery is the Bradley–Terry model. It says the probability that response i beats response j is the exponential of i's score divided by the sum of both exponentials, which collapses neatly into a sigmoid applied to the score difference. The lecturer pauses to rebuild the sigmoid from scratch — one over one plus e to the minus x, running from zero to one — and then makes the intuition explicit: if i is the winner, you want the input to that sigmoid to be large, which means a high score for winners and a low one for losers.

Then he does something I wish more lecturers did. Instead of announcing the loss function, he derives it. Take a dataset of pairs, assume independence, write the product of all those Bradley–Terry probabilities, and look for parameters that make the observed data most likely. Products of probabilities get numerically tiny, so take a log — which turns the product into a sum. Machine learning people prefer minimizing, so flip the sign. What you land on is the negative expected log-sigmoid of the score gap between winner and loser. An audience member had guessed binary cross-entropy beforehand, which the lecturer confirms is essentially the same idea arrived at by a different route.

The subtlety he savors: the loss is pairwise, but the resulting reward model is pointwise. Once trained, it takes one prompt and one response and emits one number. You needed pairs to train it; you don't need pairs to use it.

A few practical details follow. You want on the order of tens of thousands of pairs or more. Architecturally you can hang a classification head on a decoder-only LLM, or use an encoder-only model and project the CLS embedding — the first is what people actually do now. RewardBench gets named as the benchmark if you want to know how good your reward model is. Scores are typically defined along one dimension at a time, so you might have separate models for helpfulness, friendliness, and safety, or one holistic one. And human ratings are extremely sensitive to the instructions given to raters, so guidelines need to be as objective as you can make them or the labels come out noisy.

He gets two questions he answers a bit evasively. Is this regression or classification? His answer is essentially "it's probabilistic, and the raw score scale is arbitrary, so some renormalization happens at inference." What does the output look like concretely? Roughly, good completions land near one and bad ones around minus two or minus three, on a continuous scale. I found myself wanting a crisp answer here and not getting one.
 

PPO, Best-of-N, and DPO: Three Routes to the Same Goal

Stage two is where the lecture accelerates. The setup: take a prompt, let the LLM generate a full completion (also called a rollout), feed prompt plus completion to the frozen reward model, and use the resulting number to update the LLM. The reward model itself is not being trained anymore.

But there's a catch, and the lecturer spends real time on it. You want high rewards, but you also don't want to drift too far from where you started. He asks the room why, and collects three answers: catastrophic forgetting of everything the base model knows, overfitting to signal that's partly noise, and raw training instability. The third gets expanded into the reward-hacking explanation, which is the best analogy in the hour. Suppose your true objective is giving an informative lecture and your measurable reward is how loud the applause is. Optimize hard enough on applause and you'll start telling jokes. Reward goes up; objective unmet.

Hence PPO. The "proximal" is the whole point — stay near the base model. The objective has two pieces: a term pushing rewards up and a KL term penalizing distance from the reference model, which is your SFT checkpoint. The lecturer then detours into KL divergence itself, insisting it's a measure rather than a distance, writing out the sum of p times log p over q, and establishing non-negativity via Jensen's inequality. Zero only when the two distributions coincide.
 

Advantage, Value Functions, and the First Real Hand-Wave

Then comes a correction. We don't actually maximize reward, he says — we maximize advantage, meaning how much better an output is than what you'd expect. Subtracting that baseline reduces variance and stabilizes training. Estimating it requires a value function, which is a token-level predictor: given the prompt and a partial generation, guess the final reward if you keep going under the current policy. It's a regression head trained alongside the policy.

Here's where I'd flag a weakness. The lecturer openly declines to derive generalized advantage estimation, calls it beyond the scope of the class, and points to the GAE paper for anyone curious. That's a defensible choice for a survey course, but it means the central quantity in the PPO objective is handed to us as a black box. For the exam I'd want to know what's fair game and what isn't.
 

PPO-Clip and the Intuition Behind the Graph

The clipped variant gets a proper walkthrough. The objective is a minimum of two things: the probability ratio times the advantage, and the clipped version of that ratio times the advantage. Two traps get flagged before the intuition. Written as "L," it looks like a loss, but you maximize it. And the ratio is not a reward — it's the probability of the current policy divided by that of the previous iteration. Not the SFT model. The previous step within the RL phase, because you want each individual update to be small, not just the cumulative move.

With a positive advantage you want that ratio to grow, but not without limit, so the curve goes flat past one plus epsilon. With a negative advantage you want it to shrink, but not collapse, so it flattens on the low side too. Framed that way the formula stops being intimidating and starts looking like a deliberately boring engineering hack.

The alternative variant swaps the clip for an explicit KL penalty against the old policy. In modern LLM practice, he notes, people plug the frozen reference model into that KL term instead, and plenty of implementations mix both mechanisms. The 2017-vs-2025 mismatch is worth remembering.

His list of PPO's problems is long and worth memorizing: four sets of weights in play (policy, value function, reward model, frozen reference); a two-stage pipeline where fixing a broken reward model means redoing the downstream run; a pile of hyperparameters including beta, epsilon, and at least two inside GAE; genuine instability; exploration requirements, since you need generation diversity or you're not learning much; and a weak monitoring signal, because average reward tells you far less than the cross-entropy curves you get to watch during pre-training and SFT. He also defines on-policy versus off-policy here, and it's a clean distinction: on-policy means the model trains on its own fresh generations, which is what PPO does and SFT does not. GRPO gets a teaser, with the DeepSeekMath paper named as optional reading for next week.
 

Best-of-N: Skip the Training, Pay at Inference

Best-of-N is the escape hatch for people who have a reward model but no appetite for RL. Sample n completions, score them all, return the highest. The example scores are something like 0.8 for a good answer, minus two for a dismissive one, and something middling for a third.

Two objections come from the audience. If the underlying model is bad, all n samples are bad, and picking the best of a bad lot doesn't rescue you — the model has to be at least decent. And the real problem is cost: you've moved the work out of training and into inference, which is ruinous if you're serving real traffic. The lecturer's advice is to price both sides — training cost against projected serving volume — before choosing this route. He also notes that for pure ranking the score scale is irrelevant, since rescaling everything leaves the ordering intact; scale only starts to matter once the scores enter a loss function.
 

DPO: The Reward Model Was Hiding Inside Your Policy

At roughly (1:34), a second lecturer, Shervin, takes over for the final stretch. He opens by restating the complaints — PPO's four sets of weights, and the latency problem in Best-of-N. That latency point is sharper than the first lecturer's cost framing: even with infinite compute and perfect parallelism, you still wait for the slowest of n generations, and the distribution of a maximum shifts right. You can't buy your way out of it.

DPO, direct preference optimization, is the response. One loss function, fitted directly on preference pairs, no reward model anywhere in sight. The expression is a log-sigmoid over a scaled difference: how much the policy favors the winning completion versus the losing one, relative to what the frozen reference model does. The paper's title — and this is the memorable line of the lecture — is that your language model is secretly a reward model.

The derivation sketch is worth following even if the algebra is beyond what's tested. Start from the PPO objective with its KL penalty and its beta coefficient. Solve for the optimal policy, which comes out as a function of the reward, up to a normalizing partition function. Rearrange to express the reward as a function of that optimal policy instead. Now take that expression and drop it into the Bradley–Terry formulation from earlier. The rewards cancel out and you're left with a loss purely in terms of policies and preference data. Beta, typically around 0.1, is the same knob as before. Shervin acknowledges he's compressing something mathematically heavy, and he's not kidding.

The comparison section is balanced. DPO needs two model copies instead of four, is supervised rather than generative, and is dramatically easier to run. But he states plainly that PPO generally performs better, and explains why: DPO's supervision arrives as a fixed dataset that may not resemble what your model would actually produce, and that distribution shift is intrinsic to the method. One mitigation is to run a quick SFT pass on the preference data first, or to generate the data yourself and then rate it. He cites a benchmarking paper on the slide for the comparison without giving numbers, and since the claim "PPO beats DPO" is doing a lot of work in that section, I'd have liked to see the evidence spelled out rather than gestured at.

He also answers two questions I'd have asked. Yes, you keep one set of weights frozen and update the other. And no, you probably shouldn't swap in a different large model as your reference — the whole exercise is about nudging the distribution you already have, so starting from someone else's makes the objective incoherent.
 

The Teddy Bear, Reprised

The closing example is small and does a lot of work. Prompt: can I put my teddy bear in the washing machine? After SFT the model says no, it might get damaged, hand wash instead. Completely correct. Also blunt. The preference-tuned version says something closer to: better not to, your teddy bear could get hurt, a gentle hand wash is safer. Same facts, different manner.

That's the cleanest statement of what LLM tuning at this third stage is for. You're not teaching the model anything new. You're reshaping which parts of its existing distribution it reaches for first.

The last question from the room asks which method to actually use, and the answer is refreshingly undogmatic: it depends on your compute budget and how much babysitting you're willing to do. PPO is the ceiling and the headache. DPO gets you most of the way with a fraction of the effort. If you take one thing from Lecture 5 into the final, it's probably that trade-off — and the Bradley–Terry derivation, which was the one place the lecturer slowed all the way down.