Modern Reinforcement Learning: Deep Q Agents (PyTorch & TF2)

Explore how an agent observes states, acts, and receives rewards within an environment, using Q-learning to maximize long-term rewards, as shown in the frozen lake example.

Explore how reinforcement learning uses an agent interacting with an environment to maximize discounted rewards, formalized as a Markov decision process with states, actions, rewards, and a policy.

Explore how value functions and action value functions quantify expected future rewards under a policy, and how the Bellman equation recursively connects states to their successors toward optimal policies.

Compare model-based and model-free learning in reinforcement learning by examining the Bellman equation, the value function, and how dynamic programming vs. q-learning handle state transitions.

Explore the explore-exploit dilemma in model-free reinforcement learning, and learn how epsilon-greedy strategies manage exploration and exploitation to approximate the value function.

Explore temporal difference learning and q-learning with epsilon-greedy exploration in a tabular frozen lake setting. See bootstrapping updates, off-policy learning, and the learning curve showing rising performance.

Explore how deep neural networks overcome the limitations of tabular q-learning in continuous state spaces by approximating the action-value function with a deep q-learning approach and PyTorch.

Learn to implement a naive deep q network for q-learning in PyTorch, using a two-layer network with 128 hidden units, ReLU activation, mse loss, and epsilon-greedy decay for CartPole.

Code an agent class for deep q learning, enabling action selection, learning from experiences, and epsilon decrement over time. Implement a linear deep q network with epsilon-greedy policy in PyTorch.

Code the main loop of a naive deep q learning agent, initializing the environment, scores, and epsilon, performing epsilon-greedy actions, updating learning from transitions, and plotting learning curves.

Verify the functionality of our code by importing the function, setting the learning rate to 0.0001, and running the agent in the terminal to prepare for performance checks.

Analyze why a naive deep q network struggles in a continuous state space, showing epsilon decay, score drops, and bias from evaluating max actions with one network.

Switch to processing Atari screen images with convolutional neural networks to estimate state-action values in deep Q-learning, using 32 filters, kernels, stride, and frame stacking for motion.

Skim papers quickly to grasp the idea and implement a working prototype, then study abstract, intro, notation, and algorithmic details to translate deep Q-learning concepts into code.

Explore deep q-learning for Atari with a convolutional network, experience replay, and a periodically updated target network. Learn preprocessing, frame skipping, and epsilon-greedy training to build a model-free agent.

Learn to preprocess and stack OpenAI Gym Atari frames for deep Q-learning: grayscale and 84x84 resizing, max of two frames, and four-frame stacking with action repetition.

Preprocess OpenAI Gym Atari screen observations by converting to grayscale, resizing with cv2, reshaping and scaling pixels, and returning a new observation wrapper in PyTorch to prepare for stacked frames.

Learn to stack preprocessed Atari frames with an observation wrapper, maintaining a maxlen stack, updating on reset and each observation, and reshape to a numpy array for deep Q agents.

Learn to build a gym environment by composing make, repeat action, max frame, preprocess frame, and stack frames, shaping inputs to 84x84x1 with four-frame repeats and plan memory for agent.

Add reward clipping, fire first, and no ops to the environment by passing booleans, clipping rewards in step, and performing no ops then a fire action in reset.

Design and implement a reusable replay memory to store states, actions, rewards, terminal flags, and target values for the loss function; sample uniformly without duplicates and support arbitrary input shapes.

Code a deep Q-network in PyTorch with three convolutional layers and two fully connected layers, compute input size dynamically, and enable rmsprop training, mse loss, and model checkpointing.

Code the dqn agent constructor by wiring the online and target networks, replay memory, epsilon-greedy action selection, weight copying from online to target, and model checkpointing with descriptive network naming.

Implement epsilon-greedy action selection by deciding between a greedy argmax from the deep Q network and a random action, with proper input shaping for the network and gym compatibility.

Manage the agent's memory by storing transitions, sampling to PyTorch tensors, replace the target network when due, perform epsilon decrement, and provide save/load checkpoints for the deep Q agent.

Choose when to learn based on memory batch size, sample memories, compute predicted vs target Q values using the target network, apply done masks, backpropagate, and update epsilon.

Code the deep Q network agent and its main loop to train for 500 Pong games, track the running 100-game score average, save best models, and plot the learning curve.

Analyze how deep q-learning overestimates values and how double q-learning reduces bias by decoupling action selection from evaluation, using target and online networks, replay memory, and standard architectures.

Code a double deep q-learning agent by adapting the deep q-learning implementation, focusing on the target value calculation and update rule, and analyze learning with epsilon-greedy.

Explore the dueling network architecture for model-free reinforcement learning, with two streams for state value and action advantage sharing a convolutional backbone, boosting policy evaluation and Atari 2600 performance.

Code a dueling deep q learning agent by combining value and advantage streams (subtracting the mean) for q values, and modify action selection and learning to use this structure.

Implement a dueling double deep q-learning agent by combining value and advantage streams and applying the double q-learning update in the learn function, then evaluate on Pong.

Master a flexible command line interface using argparse to pass hyperparameters to your agent constructor, with optional arguments, defaults, help strings, and parse_args for rapid model testing.

Consolidate the codebase with a base agent class and derive all algorithm-specific agents, centralizing choose action, learn, and memory logic for maximum extensibility in deep Q learning.

Test a trained dqn agent greedily and visualize its real-time play. Save mp4 videos with gym wrappers to monitor and store outputs for later gif conversion.

Summarize the course journey from reinforcement learning basics to deep q-learning, double q-learning, and dueling architectures, highlighting Markov decision processes, Bellman equations, and practical code and Atari applications.

Explore next steps after deep Q-learning by comparing action value approximation to policy gradient methods for continuous actions, and find further learning resources on YouTube.

Compare tensorflow 2 and pytorch for deep q agents, covering keras models, call function, channels last vs first, model compile, save/load, and gradient tape training.

learn to implement a TensorFlow 2 deep Q network class using Keras, with conv 2d and dense layers, channels first, and an agent-level checkpointing and optimizer setup.

Code the deep Q-learning agent in TensorFlow 2 by implementing tensor-based memory sampling, target network updates, gradient tape training, and save/load models for a robust reinforcement learning workflow.

Enable memory growth for all GPUs in TensorFlow 2, monitor GPU usage with Nvidia SMI, and demonstrate a deep Q learning agent learning from about 200 games.

Implement a TensorFlow 2 double q-learning agent by updating the learn function to use q-values from both the online and target networks. Pong results show learning after about 100 games.

Implement a dueling deep q network in TensorFlow 2 by adding value and advantage outputs, updating the agent's action selection and learning step, and confirming Pong learning.

Explore coding the dueling double deep q-learning agent in TensorFlow 2, incorporating the advantage function and the value estimate, target calculations, and gradient-tape learning.