Reinforcement Learning: The Complete Course in 2022

Explore a concise robotics timeline from early automated devices to fourth generation robots, and examine how automatic control enables perception, interaction, and intelligent industrial and social robotics.

The lecture continues cart pole reinforcement learning, detailing observation, environment, left-right actions, rewards, and episodic interactions.

Explore reinforcement learning basics, environments, and how to apply a cartpole control using plus or minus one force to swing up and stabilize.

Demonstrate policy gradient concepts by running episodes, computing total rewards, and applying gradient updates with a learning rate of 0.001.

Develop a policy gradient approach by computing total and average rewards over episodes, plot performance, and update the policy via stochastic gradient using episode rewards and action probabilities.

Simulate Atari environments part 2 shows how the environment step maps actions to game moves and how render updates the observation. The agent controls a spaceship to shoot aliens.

Introductory session presents the Markov decision process and dynamic programming, guiding practical reinforcement learning with policy evaluation, value iteration, policy iteration, and the frozen environment example.

Explore the Markov property, build a transition matrix, and compute state distributions from an initial distribution using a study and sleep example.

Demonstrate how a Markov chain updates its state distribution across iterations, turning initial probabilities into evolving values like 0.52 and 0.48 and showing convergence.

Introduce MDP concepts, transition matrices, actions (a0 and a1), and a reward function, illustrating how to derive the optimal policy and its value function, using matrix inversion.

Explore reinforcement learning with MDP implementation, experimenting with gamma values 0, 0.9, and 0.99, and examine how rewards, the transition matrix, and the value function change under the optimal policy.

Simulate the frozen lake environment on a discrete grid and explore its optimal policy. Use gym to create, reset, render, and step through episodes, noting random-policy rewards and slip effects.

Simulate the frozen lake environment by applying a 16-action policy over 16 states, illustrating movement, stochastic outcomes, and episode rewards, while examining the transformation matrix and rewards.

Explore solving an mdp with a policy iteration algorithm, covering policy evaluation and policy improvements via Bellman equations, and apply it to the frozen lake environment using gym and torch.

Explore deriving the optimal policy and values for a coin-flipping gamble problem using value iteration, comparing it with policy iteration, and implementing the environment for convergence.

Explore solving the coin-flipping gamble environment with policy iteration and value iteration, deriving optimal values and policies via the Bellman optimality equation, and compare efficiency with different action counts.

Explore solving the coin-flipping gamble problem by policy iteration, comparing it with value iteration, using the Belmont optimality equation to maximize winning probability and evaluate policy values.

Explore model-free reinforcement learning with Monte Carlo methods that require no prior environment knowledge, estimating value and action-value, and applying epsilon-greedy policy and weighted importance sampling on blackjack.

Learn Monte Carlo policy evaluation for estimating value functions, comparing model-based and model-free approaches, and applying first-visit and every-visit predictions in the frozen lake environment.

Implement Monte Carlo policy evaluation in reinforcement learning by coding an episode-based framework that tracks rewards, states, and actions to estimate an optimal policy.

Explore Monte Carlo policy evaluation by implementing every-visit value predictions, updating value functions from episode returns, and applying model-free reinforcement learning concepts.

Explore blackjack with Monte Carlo prediction to evaluate policies and prepare for Monte Carlo control, using episodes, actions, and game rules to build optimal strategies.

Advance the blackjack project by implementing a Monte Carlo prediction-based value function, and build utility and plotting code to visualize state values across scenarios.

This lecture demonstrates implementing on-policy Monte Carlo control, updating q-values and value functions across episodes, evaluating and improving the policy, and visualizing results, including exploring start and first-visit updates.

Learn off-policy Monte Carlo control by using a behavior policy for exploration to improve a target policy through importance sampling, enabling policy evaluation and learning from episodes.

Learn to implement off-policy Monte Carlo control incrementally to update the Q function on the fly. The approach reduces memory usage and improves scalability.

Explore off-policy Monte Carlo control, using a behavior policy to generate episodes, evaluate a target policy with first-visit Monte Carlo, and incrementally update Q to obtain the optimal policy.

Monte Carlo control with weighted importance sampling, compare it to ordinary sampling, and reduce variance by using weighted returns from a behavior policy to optimize the target policy.

Explore how Monte Carlo control uses weighted importance sampling to scale returns and achieve lower variance than ordinary importance sampling.

The lecture explains weighted importance sampling for Monte Carlo control, showing variance reduction and a 100,000-episode simulation to estimate win and loss probabilities under the optimal policy.

Learn to set up the cliff walking environment playground, explore the gym-like observation and action spaces, and implement reset, render, and step workflows to test an agent.

Explore the cliff walking environment with a step-by-step playground walkthrough using a grid world, illustrating agent moves, resets, and rewards in reinforcement learning.

Explore the windy gridworld environment, where an external force moves the agent and complicates policy learning. Examine transition dynamics, negative one rewards, and the temperature difference method.

Implement a windy gridworld environment by setting up wind, building transition and reward matrices, computing the initial state distribution, and rendering the environment for visualization.

Explore double q-learning by updating one of two q-functions chosen at random to reduce overestimation. Observe how averaging the two q-functions guides policy convergence across episodes.

Explore bandit strategies in gym part 3 by comparing epsilon-greedy and softmax exploration, using temperature to bias arm selection by average reward during initial exploration rounds.

Explore the exploration-exploitation dilemma in reinforcement learning bandits with upper confidence bound methods, learning to select arms by confidence intervals and updating rewards to maximize long-term performance.

Explore Hobson sampling, a probabilistic bandit method using priors to balance exploration and exploitation by sampling from arm distributions, pulling the best arm, and updating alpha and delta after outcomes.

Explore how multi-armed bandit methods apply to real-world use cases beyond A/B testing, optimizing landing pages and online ads by balancing exploration and exploitation to maximize conversions.

Apply a multi-armed bandit approach to select the best advertisement banner by using an epsilon-greedy exploration strategy, maximizing the average reward across five ad panels.

Explore reinforcement learning strategies with the epsilon-greedy method and probability-based exploration, selecting random actions with epsilon and the best action otherwise, while weighting by average reward to learn value estimates.

Learn to solve internet advertising problems with contextual bandits, focusing on two states, actions, and rewards. Compare contextual bandits with broader reinforcement learning and explore practical ad experiments.

Explore a deep q-networks implementation, set up the environment, and run a training loop that updates a neural network with q-values and target values using epsilon-greedy exploration.

This lecture walks through implementing double deep Q-networks, setting up the environment, building the replay memory and target network, and configuring epsilon-based training with periodic target updates.

Implement dueling deep q-networks by building the environment and a neural network with experience memory. Learn training with rewards per episode and an epsilon-based policy using q-values and advantages.

Explore the dueling deep q-nets architecture, which combines a value estimate with an action advantage to improve learning robustness in reinforcement learning.

Demonstrates implementing reinforce with a baseline by building a policy network and a value network, computing advantages, and updating network weights via backpropagation in an environment.

Explore cliff walking in a gym environment using the actor-critic algorithm, with long episodes and -100 cliff penalties, and learn one-hot state encoding for the network.

Explore solving the mountain car environment with an advantage actor-critic method for continuous actions, using a Gaussian action distribution, a policy network, and value estimation.

Learn to solve the mountain car environment using reinforcement learning by building a modular model with an actor and critic components and training a policy network to maximize rewards.

Solve the mountain car continuous control task with a Gaussian policy network that outputs mean, variance, and value, and train with actor-critic loss and clipped actions.