
Explore how reinforcement learning uses an agent interacting with an environment to maximize discounted rewards, formalized as a Markov decision process with states, actions, rewards, and a policy.
Compare model-based and model-free learning in reinforcement learning by examining the Bellman equation, the value function, and how dynamic programming vs. q-learning handle state transitions.
Explore how deep neural networks overcome the limitations of tabular q-learning in continuous state spaces by approximating the action-value function with a deep q-learning approach and PyTorch.
Learn to preprocess and stack OpenAI Gym Atari frames for deep Q-learning: grayscale and 84x84 resizing, max of two frames, and four-frame stacking with action repetition.
Preprocess OpenAI Gym Atari screen observations by converting to grayscale, resizing with cv2, reshaping and scaling pixels, and returning a new observation wrapper in PyTorch to prepare for stacked frames.
Learn to build a gym environment by composing make, repeat action, max frame, preprocess frame, and stack frames, shaping inputs to 84x84x1 with four-frame repeats and plan memory for agent.
Add reward clipping, fire first, and no ops to the environment by passing booleans, clipping rewards in step, and performing no ops then a fire action in reset.
Code the dqn agent constructor by wiring the online and target networks, replay memory, epsilon-greedy action selection, weight copying from online to target, and model checkpointing with descriptive network naming.
Explore the dueling network architecture for model-free reinforcement learning, with two streams for state value and action advantage sharing a convolutional backbone, boosting policy evaluation and Atari 2600 performance.
Implement a dueling double deep q-learning agent by combining value and advantage streams and applying the double q-learning update in the learn function, then evaluate on Pong.
Explore next steps after deep Q-learning by comparing action value approximation to policy gradient methods for continuous actions, and find further learning resources on YouTube.
Compare tensorflow 2 and pytorch for deep q agents, covering keras models, call function, channels last vs first, model compile, save/load, and gradient tape training.
In this complete deep reinforcement learning course you will learn a repeatable framework for reading and implementing deep reinforcement learning research papers. You will read the original papers that introduced the Deep Q learning, Double Deep Q learning, and Dueling Deep Q learning algorithms. You will then learn how to implement these in pythonic and concise PyTorch and Tensorflow 2 code, that can be extended to include any future deep Q learning algorithms. These algorithms will be used to solve a variety of environments from the Open AI gym's Atari library, including Pong, Breakout, and Bankheist.
You will learn the key to making these Deep Q Learning algorithms work, which is how to modify the Open AI Gym's Atari library to meet the specifications of the original Deep Q Learning papers. You will learn how to:
Repeat actions to reduce computational overhead
Rescale the Atari screen images to increase efficiency
Stack frames to give the Deep Q agent a sense of motion
Evaluate the Deep Q agent's performance with random no-ops to deal with model over training
Clip rewards to enable the Deep Q learning agent to generalize across Atari games with different score scales
If you do not have prior experience in reinforcement or deep reinforcement learning, that's no problem. Included in the course is a complete and concise course on the fundamentals of reinforcement learning. The introductory course in reinforcement learning will be taught in the context of solving the Frozen Lake environment from the Open AI Gym.
We will cover:
Markov decision processes
Temporal difference learning
The original Q learning algorithm
How to solve the Bellman equation
Value functions and action value functions
Model free vs. model based reinforcement learning
Solutions to the explore-exploit dilemma, including optimistic initial values and epsilon-greedy action selection
Also included is a mini course in deep learning using the PyTorch framework. This is geared for students who are familiar with the basic concepts of deep learning, but not the specifics, or those who are comfortable with deep learning in another framework, such as Tensorflow or Keras. You will learn how to code a deep neural network in Pytorch as well as how convolutional neural networks function. This will be put to use in implementing a naive Deep Q learning agent to solve the Cartpole problem from the Open AI gym.