Reinforcement Learning (English): Master the Art of RL

Explore the fundamentals of reinforcement learning, contrast it with supervised learning, and introduce the action, reward, environment, and agent framework, including states, gym environment, and the policy, value, and model.

Explore reinforcement learning as the science of sequential decision making driven by reward and penalty, from episodic sparse rewards to credit assignment, online and curriculum learning.

Reinforcement learning trains an agent to maximize long-term rewards through sequential decisions in an online environment, guided by rewards, penalties, and reward shaping.

Discover the four core reinforcement learning elements—agent, environment, action, and reward—and how sequential triplets of observation, reward, and action form a history that guides learning in dynamic environments.

Explore how rewards guide reinforcement learning, from immediate and intermediate r_t to cumulative G_t across an episode. Learn reward shaping and the objective to maximize long-term reward.

Contrast reinforcement learning with supervised learning by highlighting online learning signals and delayed, sparse rewards. Show how RL uses dynamic, action-dependent data and credit assignment, unlike static IID data.

Define the state as a function of history, distinguishing environment state, agent state, and observations. Explain how the Markov assumption underpins fully and partially observable MDPs.

Explore how environments, agents, actions, rewards, and observations drive reinforcement learning across self-driving cars, chess, a maze, street fighter, and Mario, with state value and policy guiding actions.

Master OpenAI gym environments as a testing ground for reinforcement learning, exploring observations, rewards, done signals, and wrappers that shape rendering and reward processing.

Explore the RL agent brain, focusing on the policy, value function, and model, and learn how observations and rewards shape actions to maximize cumulative reward.

A policy maps states to actions, with deterministic or stochastic forms that provide a probability distribution over actions conditioned on the state, summing to one.

The value function predicts the expected discounted cumulative reward from a state. Follow a given policy and account for randomness in policy and environment.

Explore the environment model, including the state transition model, and reward model, and how stochastic dynamics and expectation shape the value function and cumulative reward.

Explore rl agents taxonomy, contrasting policy-based, value-based, and actor-critic designs, and explain how policy, value functions, model free versus model based, and the exploration-exploitation trade-off shape learning.

Explore the prediction versus control framework in reinforcement learning, evaluating a given policy with the value function and iteratively improving toward the optimal policy for maximum cumulative reward.

Explore the basics of Markov decision processes, from Markov chains to Markov reward processes, and evaluate policies with the value function using dynamic programming for control.

Define the Markov property and Markov process, showing how the current state summarizes history to predict the next state; illustrate with traffic lights and weather transitions and their transition matrices.

Examine the Markov reward process by introducing an immediate reward function, compute expected rewards with the transition matrix, and apply the Bellman equation and gamma to evaluate state values.

Explore the Markov decision process by introducing actions to the Markov reward framework. Define states, actions, transitions, and rewards, and study stochastic and deterministic policies for optimization.

Explore policy evaluation in reinforcement learning by deriving the Bellman prediction equation for MDPs and evaluating v_pi(s) and Q_pi(s,a) under a stochastic policy.

Derive the Bellman equations for V and Q, reveal their recursive relations, and show how Q-learning uses these updates under a stochastic policy to solve MDPs.

Explore how to find the optimal policy in reinforcement learning using Bellman optimality with max over actions, and implement value and policy iteration.

Explore the reinforcement learning solution space for MDPs, distinguishing planning from learning. Learn about model-based and model-free methods, including dynamic programming, TD, MC, SARSA, Q-learning, and the Bellman optimality framework.

Explore planning with dynamic programming in reinforcement learning, from exhaustive tree search and look-ahead planning to caching solutions and Bellman equations.

Apply dynamic programming policy evaluation in a grid world, using the Bellman expectation equation under a uniform random policy to iteratively compute the value function toward terminal states.

Use policy iteration with dynamic programming to find the optimal policy by evaluating policies, deriving V or Q, and greedily improving via argmax; this contrasts with value iteration.

Apply value iteration in dynamic programming to maximize rewards on a gridworld using the Bellman optimality equation. Compare with policy iteration and discuss model-based versus model-free learning.

Monte Carlo sampling estimates value functions in model-free prediction by averaging returns from trajectories, using first-visit or every-visit counts across states. It overcomes the need for an explicit environment model.

Explore temporal-difference learning for prediction in reinforcement learning, using one-step lookahead and bootstrapping to update the value function with td targets and errors, contrasting Monte Carlo and dynamic programming.

Explore how TD lambda unifies one-step td and monte carlo by weighting n-step returns with lambda, enabling a spectrum between td zero and full monte carlo.

Explore model-free control with Monte Carlo policy iteration, evaluating policies and improving them via Q estimates and epsilon-greedy exploration, aided by TD methods for faster learning.

Apply td model-free methods to control using one-step returns and q-value updates. Combine with epsilon-greedy policy, sarsa, and policy iteration, referencing q(s,a) and td targets.

Explore on-policy versus off-policy learning in model-free reinforcement learning, where a behavior policy acts while a separate target policy learns, enabling learning from human actions and safer exploration.

Q-learning is the off-policy variant of Sirsa, using an epsilon-greedy behavior policy and a greedy target policy derived from the q-function, updating by max over actions.

This lecture contrasts planning and model-free methods in reinforcement learning, detailing dynamic programming, TD learning, policy evaluation, and Q-learning with Bellman equations and value versus policy iteration.

Investigate deep reinforcement learning, combining deep learning with value function approximation for high dimensional problems, review DQN and policy gradient methods, including actor-critic approaches.

Explore why tabular reinforcement learning struggles with large or continuous state spaces, and how neural networks embed high-dimensional observations into low-dimensional features for scalable value and policy estimation.

Transform high-dimensional unstructured inputs into low-dimensional feature embeddings with deep neural networks, and use an encoder-decoder design and gradient-based optimization to close the semantic gap and produce accurate outputs.

Apply neural networks to value function approximation in reinforcement learning, using a convnet encoder to convert high-dimensional game frames into a compact state vector for Q-value evaluation of discrete actions.

Explore deep neural network policies for reinforcement learning, including feature extraction with encoders, Q and V function approximations, and diverse policies from convnets to LSTMs.

Train a value function approximation with a neural network, using gradient descent and mean squared error, from a simple linear Q function to td and Monte Carlo targets for prediction.

Explore how deep q-networks enable stable deep reinforcement learning by using experience replay, fixed q targets, and shuffled minibatches for off-policy training.

Explore deep Q networks (DQN) on Atari games with high dimensional color frames and discrete actions. Use Keras-RL and TF-Agents with experience replay and fixed Q targets.

Explore policy gradient methods that enable continuous actions in deep reinforcement learning, covering actor-critic approaches like A2C, A3C, TRPO, PPO, DDPG, trained with stable baselines.

Compare value-based and policy-based reinforcement learning, explain actor-critic as a hybrid that uses an actor (policy) and a critic (value), and introduce policy gradient methods within policy-based and actor-critic families.

Directly parameterize policy with a neural network and optimize its parameters via policy gradients to maximize expected reward, using the log-derivative trick and Monte Carlo or td for sampling.

Explore episodic returns to approximate the Q function with Monte-Carlo policy gradients, the reinforce algorithm. Update thetas via gradient ascent on expected final cumulative reward.

Explore actor-critic methods in deep reinforcement learning, combining policy gradients with a critic that estimates q values via td learning, updating theta and w parameters for improved rewards.

Explore the A2C algorithm, introducing an advantage term based on the state value function baseline and apply temporal difference methods to update policy gradients.

Explore asynchronous advantage actor-critic learning by running multiple agents in parallel environments, with parallel workers and a global parameter server aggregating updates to train STC and A3C networks.

Explore trusted region policy optimization, which uses kl divergence to constrain policy updates and stabilize policy gradient methods, including actor-critic algorithms, with proximal policy approximation next.

PPO blends TRPO and A2C to stabilize policy updates via a theta ratio between new and old policies, with clipping in [1-epsilon,1+epsilon] and an advantage-based objective, epsilon around 0.1–0.2.

Master deep deterministic policy gradient (ddpg) to train a policy and q-function end-to-end with deep networks, using exact actions and exploration noise in model-free reinforcement learning.

Explore the stable baselines library, offering out-of-the-box, stable reinforcement learning algorithms like DQN and PPO, integrated with gym and vectorized environments for efficient training and evaluation.

Explore Atari games with stable-baselines and Optuna, running ten hyperparameter trials of five parameters with 1,000-step agents to select the best for a 50,000-step training, comparing random and PPO performance.

Use stable baselines to train a Mario agent in a Gem environment, pre-processing observations with grayscale frames stacked four times to drive the ppo policy and tune hyperparameters.

Explore a street fighter reinforcement learning example using stable-baselines, with reward shaping, delta rewards, and frame-delta preprocessing to grayscale 84 by 84 to capture motion, via gym retro.

Explore model-based reinforcement learning and how planning compares to model-free methods. Learn how to build and use environment models for sample-based planning and the Dyna framework.

Learn model-based reinforcement learning by building an environment model through interaction, then plan with dynamic programming or use model-free methods, including policy and value function learning, via simulation.

Learn to build a model for reinforcement learning by predicting next state transitions and immediate rewards in an MDP using supervised learning and function approximation.

Learn to build sample based planning by learning a model via supervised learning and using dynamic programming to plan without real environment interaction.

Discover the dynamic architecture that blends model-based planning with model-free learning, using real and simulated experience to update value and policy via Dyna-Q, Q-learning, Sarsa, and Monte Carlo.

Define reinforcement learning, contrast it with supervised learning, and outline ingredients: agent, reward, environment, and actions. Clarify environment, state, and agent state concepts, including fully observable and partially observable distinctions.