Introduction to Reinforcement Learning (RL)

Review the deep q-learning approach to Atari games from the DQN paper, replacing value tables with a single neural network that predicts q-values from raw frames using experience replay.

Implement deep q-learning from scratch using PyTorch and numpy, with a replay buffer, epsilon-greedy policy, and a gym breakout environment, following the pseudocode step by step.

Demonstrates implementing deep q-network and PPO with stable-baselines, using a self-contained replay buffer, a convolutional neural network for 84x84x4 observations, and huber loss for training.

Finish implementing YJ with an epoch-based loop, computing current Q values from the Q network and target from rewards with gamma 0.99, using max next-state Q for non-terminal steps.

The lecture outlines implementing DQN training from scratch, including replay buffer sampling, four-frame update frequency, reward clipping, frame stacking, randomized initial steps, and progress bar for breakout using atari-style environments.

Implement a DQN with a CNN in PyTorch, building conv blocks from 4-channel 84x84 inputs to 16 and 32 filters, then flatten to 256 units and output per-action Q-values.

Save the q-network to cpu when total rewards surpass the max reward, update the max reward, and reuse the model to let the agent play in a new environment.

Comparing epsilon values in breakout training, the lecture shows 0.01 maintains comparable or better average rewards than 0.1, and discusses testing cadence and saving the evolving model.

Explore testing a DQN by loading a trained model, running deterministic episodes in an environment, measuring rewards, and recording videos to assess performance.

watching a reinforcement learning agent play for four minutes, it learns to dig tunnels, earns rewards, and achieves a final total reward of 264, before moving to part two.

The lecture reviews the deep q-network approach with experience replay and a target network, learning policies from pixel inputs and surpassing human performance across 49 games.

implement a target dqn from scratch by initializing a target network with the same weights as the q-network, using state dicts, and updating every 10,000 steps during training.

Explore the pseudocode for asynchronous advantage actor-critic methods, detailing shared and per-thread parameters, n-step returns, policy and value updates, and entropy regularization.

Explore implementing a synchronous A3C training loop from scratch with multiple environments, an actor-critic neural network, and stable action sampling using logit-based categorical distributions.

Compute log probabilities and entropy for actions using a categorical distribution. Fill buffers for rewards, state values, and log probabilities across environments, then apply an actor-critic loop.

Apply reward clipping and environment resets in the Atari domain while introducing a linear learning rate scheduler from start factor 1 to end factor 0.

Finalize the RL codebase by adding visualization for multi-environment rewards, updating actor-critic components, and preparing the base for PPO, with iterative plotting and model saving.

Implement an environment class for parallel agents in reinforcement learning, including len, reset, step, observation handling, rewards, done flag, and life tracking across multiple actors.

Implement from scratch by adapting a DQN module, apply a forward pass on input x divided by 255, use tanh activations, and generate actor logits and state value.

implement a from-scratch reinforcement learning setup by creating environments, eight actors, and wiring action space and step calls, while debugging type errors and preparing training toward proximal policy optimization.

Apply proximal policy optimization concepts to RL training, fix gradient handling and testing scripts, and assess agent performance through average rewards and deterministic vs stochastic policies.

Implement the second part of the RL algorithm from scratch with PyTorch, iterating over three epochs, using a data loader for mini-batches, and applying the clipped surrogate objective.

Configure the data loader with batch size and shuffle, then apply gradient clipping and clipping-based critic losses for stable reinforcement learning training.