About the Course
This course covers theory-forward advanced machine learning with a focus on generalization (PAC/VC/Rademacher/stability), optimization for ML, uncertainty with conditional guarantees (including multicalibration), and learning-in-games / online learning (e.g., multiplicative weights, OCO, minimax, swap regret).
- Prerequisites: Equivalent to UW ML 446/546 (intro ML, probability, linear algebra, optimization basics).
- Audience: Advanced undergraduates and graduate students.
- Modality: 2 lectures/week.
- Schedule: Tue/Thu 11:30–12:50 in JHN 102, starting Thu Sep 25, 2025. Dates account for campus holidays (Yom Kippur, Veterans Day, Thanksgiving).
Lectures
Two 80-minute lectures per week. Lecture notes will be linked as they are posted.
| # | Date | Topic | Lecture Notes | Resources | Posted |
|---|---|---|---|---|---|
| 1 | Thu Sep 25 | Course logistics overview, introduction to uncertainty, and (if time) learning theory basics | Notes | L1 (25sp) • L2 (25sp) • UML Ch. 2–3 | — |
| 2 | Tue Sep 30 | Learning theory basics: PAC learning and ERM | Notes | L2 (25sp) • UML Ch. 2–3 | HW1 out |
| 3 | Thu Oct 2 | No class (Yom Kippur) | — | — | — |
| 3 | Tue Oct 7 | Uniform convergence and concentration | Notes | L3 (25sp) • L4 (25sp) • UML Ch. 4, App B • Concentration Inequalities Handout | — |
| 4 | Thu Oct 9 | No-free-lunch; VC dimension I | Notes | L5 (25sp) • L6 (25sp) • UML Ch. 5–6 | — |
| 5 | Tue Oct 14 | VC dimension II | Notes | L6 (25sp) • UML Ch. 6 | Notes |
| 6 | Thu Oct 16 | Regularization and algorithmic stability | Notes | L8 (25sp) • UML Ch. 12–13 | — |
| 7 | Tue Oct 21 | Why Smooth, Convex loss minimization generalizes | Notes | — | — |
| 8 | Thu Oct 23 | Mean and quantile consistency | Notes | Uncertainty Notes Ch. 2 (Means & Quantiles) | — |
| 9 | Tue Oct 28 | Marginal Quantile consistency | Notes | Uncertainty Notes Ch. 2 §2.2 (Quantiles) | — |
| 10 | Thu Oct 30 | Online marginal means/quantiles, introduction to calibration | Notes | Uncertainty Notes Ch. 2 (Means & Quantiles) • Uncertainty Notes Ch. 3 §3.1–3.2 (Calibration basics) | — |
| 11 | Tue Nov 4 | Online calibration guarantees | Notes | Uncertainty Notes Ch. 3 §3.4 (Sequential Calibration) | — |
| 12 | Thu Nov 6 | Continuation of online calibration, introduction to group guarantees | Notes | Uncertainty Notes Ch. 4 (Multigroup Guarantees) | Project Milestone Due |
| — | Tue Nov 11 | No class (Veterans Day) | — | — | — |
| 13 | Thu Nov 13 | TBA | Notes | — | — |
| 14 | Tue Nov 18 | TBA | — | — | — |
| 15 | Thu Nov 20 | TBA | — | — | — |
| 16 | Tue Nov 25 | TBA | — | — | — |
| — | Thu Nov 27 | No class (Thanksgiving) | — | — | — |
| 17 | Tue Dec 2 | TBA | — | — | — |
| 18 | Thu Dec 4 | TBA | — | — | — |
Office Hours
- Instructor: Jamie Morgenstern — Tue 12:50-1:50pm (right after class), Gates 315
- TA: Rachel Hong — Thu 2-3pm, Gates 131
Emails: hongrach@cs.washington.edu; hongrach@uw.edu - TA: Bernie Zhu — Thu 4-5pm, Allen 220
Emails: haozhu@cs.washington.edu; haozhu@uw.edu
Assignments
- Homework 1: ML Theory Basics — PAC/VC, uniform convergence, Rademacher/stability. Release: W2 • Due: Fri Oct 17, 2025 at 11:59pm
Homework 1 PDF • LaTeX Source (ZIP)
Note: This homework includes topics we will cover in upcoming lectures. You are encouraged to start early and work on problems as we cover the material in class. - Homework 2: Calibration — Marginal mean/quantile consistency and basic theoretical calibration. Release: W4 • Due: Tue Nov 25, 2025 at 11:59pm
Homework 2 PDF • LaTeX Source
Supplemental: Comparing Calibration Error Metrics (ECE, SCE, MCE)
Reading Assignments (Graduate)
Graduate students will read and write summary posts on Ed Discussion for 4 contemporary papers on uncertainty estimation. Choose 4 papers from the list below (or propose alternatives with instructor approval).
Assignment Format
For each paper, post a summary on Ed Discussion (100-300 words, approximately half a page) that includes:
- Main contributions and key ideas
- How the work relates to course topics (e.g., calibration, conformal prediction, distribution shift)
- One strength and one limitation or open question
Paper List: Uncertainty Estimation (2021-2024)
- Conformal Prediction: Angelopoulos & Bates (2021). "A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification." arXiv:2107.07511
- Multicalibration: Gopalan et al. (2022). "Low-Degree Multicalibration." COLT 2022. arXiv:2203.01255
- Distribution Shift: Koh et al. (2021). "WILDS: A Benchmark of in-the-Wild Distribution Shifts." ICML 2021. arXiv:2012.07421
- Uncertainty in Deep Learning: Gawlikowski et al. (2023). "A Survey of Uncertainty in Deep Neural Networks." Artificial Intelligence Review. arXiv:2107.03342
- Calibration Methods: Kumar et al. (2023). "Calibrated Ensembles: A Simple Way to Mitigate Overconfidence in Deep Learning." ICLR 2023. OpenReview
- Selective Prediction: Geifman & El-Yaniv (2019). "SelectiveNet: A Deep Neural Network with a Reject Option." ICML 2019. arXiv:1901.09192
- Post-hoc Calibration: Minderer et al. (2021). "Revisiting the Calibration of Modern Neural Networks." NeurIPS 2021. arXiv:2106.07998
- Fairness & Uncertainty: Jung et al. (2022). "Moment Multicalibration for Uncertainty Estimation." COLT 2022. arXiv:2206.04204
Timeline: Post one summary every 2 weeks starting Week 5.
Projects + Grading
The final project can be a reproduction study, original empirical system, theory survey with extension, or an uncertainty/online-learning algorithm implementation with empirical evaluation. Teams should be 1--4 students.
Looking for teammates? Use the Team Finder thread on Ed Discussion to connect with other students.
- Proposal (10%) — Due Tue Oct 21, 2025
Project Proposal Guidelines (PDF) - Milestone (25%) — Due Thu Nov 6, 2025
- Final Report + Presentation (65%) — Due during the official final exam slot (see Final Exam below)
Course Grading
- Undergraduates: HW1 20%, HW2 20%, Participation 10%, Project 50%
- Graduates: HW1 20%, HW2 20%, Reading 10%, Participation 10%, Project 40%
Final Exam
Per the UW Seattle Autumn 2025 Final Examination Schedule, Finals Week is Dec 8–12, 2025. The last day of instruction is Fri Dec 5, 2025. The exact final time is determined by the first listed meeting day/time in the Time Schedule (for T/Th classes, it uses the Tuesday slot).
- When: Tue Dec 9, 2025 — Time per UW finals grid for T/Th classes starting at 11:30 (exact time to be posted once confirmed)
- Where: JHN 102 (unless the Registrar assigns a different room)
- What happens in the slot: Final project presentations and all remaining deliverables (final report/code) are due at the start of the slot.
- Policy: See UW Autumn 2025 Final Exam Schedule and Final Examination Guidelines.
We will post the exact time once confirmed by the Registrar’s finals grid for T/Th 11:30–12:50 classes.
Where to get help
- Discussion (Ed): Course Ed forum (join link will be posted on Day 1)
- Email: cse493s-staff@cs.washington.edu
- Office Hours: See above
- Accommodations: Please contact Disability Resources for Students (DRS)
- Public/Anonymous Posts
Questions like: “Is there a typo in the homework?”, “What does this notation mean?”, “Is this an accurate description of how this works?”. Questions that are not of a personal nature should be posted to the discussion board. - Private Posts
Questions involving your own code should be posted privately to the EdStem discussion board (not office hours). - Personal concerns
Examples: “I was in the hospital”, “Laptop was stolen”. Please email the course staff at cse493s-staff@cs.washington.edu. - Contact policy
Please direct all course-related inquiries to EdStem or cse493s-staff@cs.washington.edu. Please do not email the instructors or TAs individually. - Anonymous feedback
Submit anonymous feedback here (link will be posted when available)