Open course

Textbook

• I have been teaching a course on reinforcement learning for five years. Along with my teaching, I developed a textbook and an open course entitled "Mathematical Foundations of Reinforcement Learning".

• The PDF of the book as well as lecture notes are available on the GitHub homepage. The textbook has received 10,000+ stars on GitHub!

• Novelty: This book provides a mathematical but friendly introduction to reinforcement learning. Its content structure is also novel.

Lecture Videos

• I developed both Chinese and English lecture videos. You can check our Bilibili channel or our YouTube channel

• The lecture videos have received 1,600,000+ views over the Internet and received very good feedback! 

Below are the links to my English lecture videos.

• Overview of Reinforcement Learning in 30 Minutes

• L1: Basic Concepts (P1-State, action, policy, ...)

• L1: Basic Concepts (P2-Reward,return, Markov decision process)

• L2: Bellman Equation (P1-Motivating examples)

• L2: Bellman Equation (P2-State value)

• L2: Bellman Equation (P3-Bellman equation-Derivation)

• L2: Bellman Equation (P4-Matrix-vector form and solution)

• L2: Bellman Equation (P5-Action value)

• L3: Bellman Optimality Equation (P1-Motivating example)

• L3: Bellman Optimality Equation (P2-Optimal policy)

• L3: Bellman Optimality Equation (P3-More on BOE)

• L3: Bellman Optimality Equation (P4-Interesting properties)

• L4: Value Iteration and Policy Iteration (P1-Value iteration)

• L4: Value Iteration and Policy Iteration (P2-Policy iteration)

• L4: Value Iteration and Policy Iteration (P3-Truncated policy iteration)

• L5: Monte Carlo Learning (P1-Motivating examples)

• L5: Monte Carlo Learning (P2-MC Basic-introduction)

• L5: Monte Carlo Learning (P3-MC Basic-examples)

• L5: Monte Carlo Learning (P4-MC Exploring Starts)

• L5: Monte Carlo Learning (P5-MC Epsilon-Greedy-introduction)

• L5: Monte Carlo Learning (P6-MC Epsilon-Greedy-examples)

• L6: Stochastic Approximation and SGD (P1-Motivating example) 

• L6: Stochastic Approximation and SGD (P2-RM algorithm: introduction) 

• L6: Stochastic Approximation and SGD (P3-RM algorithm: convergence) 

• L6: Stochastic Approximation and SGD (P4-SGD algorithm: introduction) 

• L6: Stochastic Approximation and SGD (P5-SGD algorithm: examples) 

• L6: Stochastic Approximation and SGD (P6-SGD algorithm: properties) 

• L6: Stochastic Approximation and SGD (P7-SGD algorithm: comparison) 

• L7: Temporal-Difference Learning (P1-Motivating example)

• L7: Temporal-Difference Learning (P2-TD algorithm: introduction)

• L7: Temporal-Difference Learning (P3-TD algorithm: convergence)

• L7: Temporal-Difference Learning (P4-Sarsa)

• L7: Temporal-Difference Learning (P5-Expected Sarsa & n-step Sarsa)

• L7: Temporal-Difference Learning (P6-Q-learning: introduction)

• L7: Temporal-Difference Learning (P7-Q-learning: pseudo code)

• L7: Temporal-Difference Learning (P8-Unified viewpoint and summary)

• L8: Value Function Approximation (P1-Motivating example – curve fitting)

• L8: Value Function Approximation (P2-Objective function) 

• L8: Value Function Approximation (P3-Optimization algorithm)

• L8: Value Function Approximation (P4-illustrative examples and analysis)

• L8: Value Function Approximation (P5-Sarsa and Q-learning)

• L8: Value Function Approximation (P6-DQN–basic idea)

• L8: Value Function Approximation (P7-DQN – experience replay)

• L8: Value Function Approximation (P8-DQN – implementation and example)

• L9: Policy Gradient Methods (P1-Basic idea)

• L9: Policy Gradient Methods (P2-Metric 1–Average value)

• L9: Policy Gradient Methods (P3-Metric 2–Average reward)

• L9: Policy Gradient Methods (P4-Gradients of the metrics)

• L9: Policy Gradient Methods (P5-Gradient-based algorithms & REINFORCE)

• L10: Actor-Critic Methods (P1-The simplest Actor-Critic)

• L10: Actor-Critic Methods (P2-Advantage Actor-Critic)

• L10: Actor-Critic Methods (P3-Importance sampling & off-policy Actor-Critic)

• L10: Actor-Critic Methods (P4-Deterministic Actor-Critic)

• L10: Actor-Critic Methods (P5-Summary and goodbye!)

Lecture Slides

The slides for the above lecture videos can be found here: https://github.com/MathFoundationRL/Book-Mathematical-Foundation-of-Reinforcement-Learning/tree/main/Lecture%20slides