Reinforcement learning (RL) is the study of how an agent should act in an uncertain, sequential environment to maximize cumulative reward. This course develops the mathematical and algorithmic foundations of RL, with an emphasis on provably efficient methods and their theoretical guarantees. We will progress from the classical tabular setting to modern function approximation, studying both offline (batch) and online (interactive) learning paradigms along the way.
The field of RL theory has matured rapidly in recent years, yielding a rich set of tools spanning Markov decision process (MDP) theory, statistical learning, and online learning. This course will equip you with the core ideas and proof techniques that underlie this literature. By the end of the course, you will be positioned to read and contribute to research in statistical reinforcement learning.
We will cover selected topics from [AgarwalBrantleyJiangKakadeSun]:
Prerequisites: The course will make frequent references to introductory concepts of machine learning (e.g., CSE 446/546) but it is not a prerequisite. Fluency in basic concepts from linear algebra, statistics, and calculus will be assumed (see HW0). Some review materials:
The course will pull from the following textbook and course notes.
Additional resources (not used directly in the course, but useful especially for practical implementation of RL):
We will use Ed as a discussion board (you should have received an invite if registered for the course, otherwise email the instructor). We will not be using Canvas discussion board. Ed is your first resource for questions. For private or confidential questions email cse542-staff@cs.washington.edu or the instructor directly. You may also get messages to the instructor through anonymous course feedback (though, I cannot respond to you personally so this is far from ideal).
There will be 3 homeworks (each worth 20%) and a final project worth 40%. The final project involves choosing a topic in reinforcement learning and writing a summary and literature review of that area — think of it as preparing a primer for someone about to enter research in that field. Details forthcoming.
Each homework assignment will be submitted as a single PDF to Gradescope. Any code for a programming problem should come at the end of the problem, after any requested figures for the problem. You will receive an email invite once you join the course -- if not please let me know! We expect all assignments to be typeset (i.e., no photos or scans of written work). This can be done in an editor like Microsoft Word or Latex (highly recommended). There exist convenient packages for listing Python code in Latex.
Homeworks must be done individually: each student must hand in their own answers. In addition, each student must write their own code in a programming part of the assignment. It is acceptable, however, for students to collaborate in figuring out answers and helping each other solve the problems. You also must indicate on each homework with whom you collaborated. If you ever find yourself copy and pasting code / latex / whatever, you have crossed the line of the collaboration policy. You may use LLMs (e.g., ChatGPT) while completing homeworks, and you are encouraged to use them as a learning aid. However, if you use an LLM you must attach a link to the full transcript (supported by all major chatbots). This is to protect you: a transcript you are comfortable sharing is a reasonable check that the LLM helped you learn rather than did the work for you. If you find yourself copying and pasting parts of the assignment into the prompt or receiving substantial, non-boilerplate code or derivations from the LLM, you have crossed the line.
Violations of the collaboration or LLM policy are taken very seriously. If a violation is even suspected, I am obligated to report it to the Community Standards and Student Conduct office, and I will err on the side of caution. If LLM use is suspected and no transcript is attached, a report will be filed. You know the difference between an LLM helping you learn and an LLM doing it for you but if in doubt, ask on Ed!
The homework problems have been carefully chosen for their pedagogical value and hence might be similar or identical to those given out in past offerings of this course at UW, or similar courses at other schools. Using any pre-existing solutions from these sources, from the Web or other textbooks constitutes a violation of the academic integrity expected of you and is strictly prohibited.
All requests for regrading should be submitted to Gradescope directly. Office hours and in person discussions are limited solely to asking knowledge related questions, not grade related questions. If you feel that we have made an error in grading your homework, please let us know with a written explanation, and we will consider the request. Please note that regrading of a homework means the entire assignment may be regraded which may cause your grade on the entire homework set to go up or down. Regrade requests must be submitted within 7 days (24*7 hours) of the time in which grades are released.