Loading page…
THE ENGINEERING NOTEBOOK
Reinforcement Learning (RL) is a branch of machine learning in which an agent learns a policy—a strategy for choosing actions—by interacting with an environment.
At each step, the agent observes the environment, takes an action, and receives a reward and a new observation. It uses this feedback to improve its policy, with the goal of maximizing expected cumulative reward over time.
This makes RL useful for sequential decision problems, where an action can change the choices and rewards available later.
STAGE 01
What does it mean for an agent to learn from interaction?
Start with multi-armed bandits and the RL formulation, then connect policies, returns, state values, and Q-values.
STAGE 02
How can we plan when the environment dynamics are known?
Use Markov decision processes (MDPs) and Bellman equations to connect policy evaluation, policy iteration, generalized policy iteration (GPI), and value iteration.
STAGE 03
How can experience replace a model of the environment?
Build from Monte Carlo (MC) and temporal difference (TD) to multi-step TD and eligibility traces, then examine exploration, on-policy and off-policy learning, importance sampling, SARSA, and Q-learning.
STAGE 04
What changes when a value table becomes a learned function?
Introduce parameterized value functions and DQN, with a replay buffer and target network for training stability and Double DQN to address overestimation.
STAGE 05
How can we improve a policy by optimizing it directly?
Follow policy gradients and REINFORCE through baselines and advantage estimates, then study how TRPO and PPO constrain policy updates.
STAGE 06
How can a learned critic guide the actor's next update?
Combine policy and value learning with A2C and the PPO training loop, then extend to continuous control with DDPG and entropy-regularized SAC.
COURSE SOURCE NOTES
This series follows the topic scope of ECE1508: Reinforcement Learning, Fall 2025, taught by Ali Bereyhi at the University of Toronto. The stages group course topics into a suggested reading path for these notes.