LLM Evaluation — Annotated Lecture Notes & Key Takeaways

Lecture 8 is the session where the course stops building things and starts asking the uncomfortable question: how do you actually know whether any of it worked? The instructor opens by calling it one of the most important classes of the quarter, and the reasoning is simple enough that it's easy to miss how load-bearing it is — if you can't measure LLM evaluation outcomes, you don't know what to fix. Everything after that is a long march away from an ideal that doesn't scale, toward approximations that sort of work, plus a second half on agent failure modes and the benchmark ecosystem.

The class is split between two speakers. The first instructor (captions render the name as Ashin) handles roughly the first hour: human ratings, agreement statistics, reference-based metrics, and LLM-as-a-judge. Shervin takes over around the one-hour mark for agent debugging and benchmarks. The seam is visible, and the two halves have different textures — the first is conceptual and math-light, the second is a practical taxonomy.
 

Where the Lecture Starts: Humans, Subjectivity, and the Agreement Problem

The opening move is definitional, and it's more useful than it sounds. "Evaluation" turns out to be an overloaded word. It can mean output quality — coherence, factuality, relevance. It can also mean system-level properties: latency, price, uptime. The lecture explicitly narrows scope to output quality, and I appreciated the honesty there, because a lot of public argument about model quality quietly conflates the two.

The framing problem gets stated early: these models are text-to-text, so the output space is basically unrestricted. Natural language, code, math derivations. There is no single metric that covers all of it, and the lecturer doesn't pretend otherwise.

The ideal scenario is then laid out and immediately dismantled. In the ideal, every model response gets rated by a human, and you aggregate. Two problems kill it. Cost and speed are the obvious ones — asking someone to rate a thousand outputs takes a while. The subtler problem, and the one the lecture spends real time on, is that human judgment itself is fuzzy.

The example is a birthday gift query. The model answers that a teddy bear is almost always a sweet gift, pick one that feels right. Asked to rate usefulness, one rater says yes, useful, it points you in a direction. Another says no, not useful — it never says which stuffed animal, bear or elephant or giraffe. Both are defensible readings of the same response. That leads to inter-rater agreement: the degree to which your raters are aligned, which you care about because inconsistent ratings mean your evaluation signal is noise.

The fix in practice is unglamorous. You write clearer guidelines, and if agreement stays low you hold alignment sessions where raters argue through disagreements. The lecturer describes agreement as a health metric rather than a quality metric, which I thought was a genuinely clarifying distinction.

Then comes the best stretch of the first half. Raw agreement rate — the fraction of times two raters say the same thing — is shown to be close to meaningless on its own, because some agreement happens by chance. The derivation is done live on the board. With two raters, Alice and Bob, each independently saying "good" with some probability, chance agreement is P(A)P(B) + (1−P(A))(1−P(B)). Set both to 0.5 and you get 0.25 + 0.25 = 0.5. Two coin flippers agree half the time. Worse, chance agreement rises as the individual probabilities get more lopsided, so a high agreement number might just mean both raters love saying yes.

That motivates chance-corrected statistics: Cohen's kappa for two raters, with Fleiss' kappa and Krippendorff's alpha as extensions. The structure is always the same — take observed agreement, subtract what chance would have given you, normalize. Positive means you're beating randomness, 1 is perfect, negative means worse than chance. I found this part of the lecture would have benefited from a concrete numeric example with, say, 100 rated items, because the formula alone is abstract and the intuition is the whole point.
 

From Fixed References to LLM Evaluation with a Model as Judge

The next pivot is a workflow change: instead of rating every output, have humans write reference answers once for a fixed prompt set, then compare model outputs to those references automatically. You can now iterate on your model without re-hiring annotators each round.

The rule-based metrics that do this comparison get a fast tour. METEOR (metric for evaluation of translation with explicit ordering) computes a weighted F-score over matched unigrams, then multiplies by a penalty term driven by how fragmented the matches are — few long contiguous chunks is good, many scattered ones is not. BLEU (bilingual evaluation under study) is described as precision-flavored, matching n-grams in the candidate against the reference, with a brevity penalty so that very short translations can't game it. ROUGE shows up as the summarization counterpart.

The lecture's own critique of these is blunt and I think correct: the formulas look arbitrary, with hyperparameters picked by hand, and more importantly they punish stylistic variation. The demonstration is a single sentence restated three ways — a plush teddy bear can comfort a child at bedtime; soft stuffed bears help kids feel safe falling asleep; youngsters rest easier cuddling a gentle toy companion. Same meaning, almost no surface overlap, and every one of those metrics would score at least two of the three poorly. Adding synonyms and word stems helps at the margin but doesn't fix it. On top of that, correlation with human judgment is mediocre, and you still need humans to create the references in the first place.

That sets up the headline method. The intuition is that models trained on enormous corpora and tuned to human preferences already encode something like human taste, so why not use one to grade another? LLM as a judge takes the original prompt, the model's response, and a criterion, and returns a score. The term is credited to a paper from about two years before the lecture.

Two design details matter. First, the judge is asked to emit a rationale before the score, which empirically improves quality and connects back to the reasoning-material from earlier in the course — letting the model verbalize its thinking before committing. Second, and this is a nice callback, free-form output can't be parsed reliably, so the fix is constrained guided decoding from lecture 3, commercialized as structured output APIs. You define a response object with a rationale field and a score field and force the model to produce exactly that.

There are two shapes of judge. Pointwise: here is one response, is it good. Pairwise: here are two, which is better. The pairwise form doubles as a synthetic preference-data generator, which links back to the reward-model lecture.
 

Scoring factuality without a single reference

Factuality gets its own treatment because a binary good/bad call throws away too much. The worked example is a short passage about teddy bears: invented in the 1920s, named after Theodore Roosevelt, who proudly wanted to shoot a captured bear. Two errors are planted — the decade should be the 1900s, and Roosevelt refused to shoot rather than wanting to.

The procedure has stages. Decompose the passage into a list of atomic facts (the example yields four). Verify each one separately, usually in binary fashion, often using retrieval or web search as the checker. Optionally weight facts by importance — the president's attitude being less central than the naming. Then aggregate into a score, which in the example comes out at 6. The takeaway is that you get partial credit instead of a single verdict, and that the weighting scheme is yours to choose.

The lecturer is upfront that how best to do this is an open question with active papers on it. I'd flag the verification step as the load-bearing weak point: decomposing into facts and checking each one just relocates the problem into a judge you still have to trust, and if your retrieval source is wrong, your fact-check is wrong. The lecture doesn't push on that.
 

Judging Is Biased: Position, Verbosity, and Self-Enhancement

Three named biases get the most attention. Position bias: in a pairwise setup, models lean toward whichever response came first, and the standard remedy is cheap — ask A-versus-B, then B-versus-A, and take the majority. More sophisticated fixes involving position embeddings exist but aren't what you'd reach for first.

Verbosity bias: longer responses win even when they aren't better. Remedies include naming length explicitly in the guidelines, adding in-context examples, and penalizing output length. This one is acknowledged as harder to shake.

Self-enhancement bias: a model grading its own output tends to prefer it. The suggested explanation is probabilistic — if the model generated that sequence, it was by definition high-likelihood under that model, so it looks good to itself. Guidance is to use a different, ideally larger and stronger-reasoning model as judge, while conceding that this is getting hard to honor when everyone trains on similar data.

A student question during this section is worth recording: what if the judge is simply misaligned with ground truth and favors one label over another? The answer accepts it as a real additional bias and concedes the list of three is not exhaustive. That exchange is the most honest moment in the lecture about how un-rigorous this all is.

Best practices follow, and they're the practical payload: write crisp guidelines; prefer binary pass/fail over graded scales, since it's easier for both the judge and the humans you validate against; rationale before score; calibrate judge scores against a human-rated subset and run correlation analysis; use low temperature, values like 0.1 or 0.2, so results reproduce across days.

The closing warning of the first half is the part I'd underline hardest. Your judge score is a proxy for human preference. Improve the model against the proxy long enough and you can drive the proxy up while the thing you actually care about stays flat or drifts.
 

What Breaks in an Agent, Step by Step

Shervin's half opens by returning to the ReAct loop — observe, plan, act, repeating — and asking what "evaluation" even means when the thing being evaluated is a multi-step trajectory. The approach is to walk the loop and enumerate failures, using a running example: a user asks to find a bear nearby, and the available tool is find_teddy_bear.

Four failures sit at tool prediction. The model punts, failing to answer at all, often because the tool router — the recall-oriented filter that trims which function schemas get loaded into the prompt — never surfaced the right tool. Or the tool was present in the schema but the model ignored it, which points back to how tool use was taught, whether through fine-tuning or prompting. Or the model hallucinates a function that doesn't exist, calling find_bear instead of find_teddy_bear; the remedies offered range from upgrading a too-weak model to rewriting your function names, arguments, and docstrings, since those three things are the only surface the model sees. Or the model picks the right tool with wrong arguments, like passing coordinates 0,0 because nothing in context says where the user is — fixable by carrying location in context or adding a location tool that fails loudly instead of silently.

Two failures sit at execution. A tool can return an error or a wrong value; raw Python tracebacks are a poor handoff because the model may absorb the failure as its own fault and apologize vaguely instead of telling the user something actionable. Prefer structured outputs that convey what happened. And a tool can return nothing, which is genuinely dangerous for action tools — a thermostat command that returns no confirmation leaves the model free to claim success it never achieved. The specific tip is memorable: return an empty JSON rather than nothing, because an empty JSON can mean "I searched and found no bears," whereas null means nothing at all.

The seventh failure is synthesis. Everything upstream worked, the tool found a bear named Teddy one mile away, and the model still tells the user it found nothing. Causes here are weak grounding, tool output so voluminous that the useful signal drowns, and poorly shaped output — a structured object with name and distance attributes beats an undifferentiated blob.

The meta-advice is worth keeping: categorize errors methodically and fix them in groups, because otherwise every bug is its own adventure.
 

Benchmarks, Pareto Frontiers, and Why Scores Stop Being Trustworthy

The benchmark tour sorts evaluations into categories rather than ranking models. Knowledge benchmarks, typified by MMLU, span roughly 60 subjects from law to medicine and largely measure how well pre-training retained information, with four-option multiple choice chosen partly because it allows hard-coded answer extraction rather than another layer of judge error.

Reasoning benchmarks require multi-step thought. AIME is used for math, being conveniently constrained to three-digit answers. PIQA is offered as grounded physical common sense — the example being how to find something lost in a carpet, where vacuuming with a hairnet works and a solid seal does not.
Coding is justified twice over: models are used for AI-assisted coding directly, and agentic tool calls are frequently code. SWE-bench builds from real pull requests that fix an issue and add tests, so the tests give you a before/after signal on whether a proposed patch works.

Safety is the category the lecture treats most skeptically, and I think appropriately. Providers don't advertise safety scores the way they advertise reasoning scores, because safety encodes each company's own policy. HarmBench is described as covering standard harmful behavior, copyright, contextual, and multimodal cases, evaluated by a trained classifier rather than exact matching — which the lecturer notes makes it the one benchmark here that inherits classifier error. It also counts an attack as successful if the model tried, even incompetently, which implicitly separates model capability from safety.

Agent behavior gets τ-bench, spanning airline and retail domains with tools, policies, and tasks, where a separate model simulates the user because conversations can't be scripted in advance. Success is a reward computed from the resulting database state. The metric is pass^k — every one of k attempts must succeed, not just one — on the argument that reliability, not lucky success, is what makes automation usable. There's a throwaway line about deriving the formula from the earlier pass@k material and inviting students to check it at home, which is fine but left me wanting the actual algebra.

The last few minutes turn philosophical in a useful way. Benchmarks describe a profile, not a verdict; a model can be strong in one area and weak in another. Plot performance against price and the set of best-options-at-each-price forms a Pareto frontier, and you can build such a frontier for cost, safety, or context length. Contamination is the standing threat — benchmarks are only as good as the assumption the model never saw them, hence hashing, block lists, and fresh math exams. Goodhart's law gets quoted as the governing adage: when a measure becomes a target, it stops being a good measure.

The speaker closes with a personal aside about which commercial models he reaches for when coding versus when he wants something fast and cheap, immediately noting these aren't universal recommendations. Try the models yourself, he says, and there the class ends with Thanksgiving wishes.

If you take one thing from this session, it's that LLM evaluation is a stack of approximations with no clean foundation underneath — chance-corrected agreement over subjective human calls, reference metrics that can't handle paraphrase, judges with documented biases, and benchmarks that decay once they become targets. The lecture doesn't resolve that, and I don't think it's pretending to. What it does give you is vocabulary for each layer's failure mode, which is the practical version of knowing what to fix.