Advanced Reinforcement Learning: policy gradient methods

Explore reinforcement learning fundamentals by identifying five universal elements in control tasks: state, actions, rewards, agent, and environment, illustrated with chess, robotic arm, and Pac-Man.

Define the Markov decision process as a discrete time, stochastic, memoryless control framework that uses states, actions, rewards, and transition probabilities to optimize task goals.

Explore finite and infinite Markov decision processes, and episodic versus continuing tasks, with examples like a 5x5 maze and a car's continuous states.

A trajectory is the sequence of states and actions an agent experiences while moving between states; an episode starts at the initial state and ends at the final state.

Explain the difference between reward and return, and show how maximizing the long-term expected return guides action choices, noting that short-term rewards can harm long-term outcomes, illustrated by chess.

Explore how the discount factor gamma shapes reward planning in reinforcement learning, balancing immediate vs long-term gains by discounted returns in a maze task.

Explore the agent's policy, a state-to-action function, and how policies maximize the discounted sum of rewards. Differentiate stochastic and deterministic policies to choose actions.

Explore evaluating states and actions with v(s) and q(s,a). V(s) is the return from a state under a policy; q(s,a) is the return after taking an action, guiding policy optimization.

Uncover the Bellman equations for state value and Q-value, expressing expected returns as discounted rewards from successor states, enabling policy evaluation and solving control tasks in reinforcement learning.

Maximize the expected return to solve a Markov decision process by finding the optimal policy and optimal state values, using Belmont optimality equations to relate values and Q-values.

Explore Monte Carlo methods by learning from experience to estimate state-action values under a policy, using returns to update estimates and rely on law of large numbers without a model.

Apply Monte Carlo methods to solve control tasks using generalized policy iteration, evaluating and improving policies with returns and Q-values, and balancing exploration via stochastic policies or exploding start.

Implement on-policy Monte Carlo control with an epsilon-greedy policy that occasionally selects random actions; update action-value estimates by averaging returns across episodes to derive a near-optimal policy.

Temporal difference methods learn from experience to update Q-values and policy during an episode, blending Monte Carlo and dynamic programming with bootstrapping and generalized policy iteration.

Apply temporal difference methods for solving control tasks by updating a state-action value table using rewards, temporal difference error, Bellman equations, and next state estimates with a learning rate alpha.

Compare Monte Carlo and temporal difference methods by illustrating late versus immediate updates: Monte Carlo updates at episode end, while temporal difference updates after each action, enabling influence on behavior.

Learn the on-policy sarsa method with epsilon-greedy exploration and update the q-table via the temporal-difference error using the sassa elements: state, action, reward, next state, next action.

Explore off-policy q-learning with two policies: a greedy target policy and an exploratory policy that collects samples, updating the q-values via the temporal-difference rule to obtain the optimal policy.

Temporal difference learning updates value estimates during an episode, enabling faster learning than Monte Carlo methods and dynamic programming, by focusing on goal-relevant states and learning from environment samples.

Explore the family of n-step temporal-difference methods that blend Monte Carlo and TD learning, using one to multiple actual rewards and bootstrapped estimates to form step return targets.

Explore how Monte Carlo and temporal difference methods connect to n-step methods, showing how choosing different n balances Monte Carlo returns and TD updates.

Explore how the choice of n in return estimation affects bias and variance, noting that smaller n increases bias, larger n increases variance, and intermediate values perform best.

Explore how function approximators replace tabular value estimates with parameterized models for continuous state spaces. Compare linear and polynomial estimators, learning weights to achieve memory-efficient, adaptable value approximation.

Explore artificial neural networks as flexible function approximators, focusing on the basics: neurons and edges forming input, hidden, and output layers, with feedforward, fully connected networks for deep reinforcement learning.

Explore artificial neurons that aggregate weighted inputs, apply activation functions like ReLU and sigmoid, and propagate signals through hidden and output layers.

Represent a three-layer neural network in code with 3 input neurons, 6 hidden neurons, and 2 outputs to approximate the value function for a three-dimensional state with two actions.

Learn how stochastic gradient descent minimizes the neural network cost by using environment rewards, computing the gradient via backpropagation, and updating parameters to move toward minima.

Optimize a neural network to estimate state-action values by minimizing mean squared error, using rewards plus discounted next-action Q-values as targets, and examine cost surfaces with local and global minima.

Learn policy gradient methods, where a neural network defines action probabilities, producing stochastic policies that offer smoother learning and address limits of value based approaches.

Represent policies using a neural network by mapping inputs through hidden and output layers, and apply softmax to produce a probability vector over actions.

Evaluate a neural network policy’s performance to compare and select the best policy, then optimize parameters via stochastic gradient ascent using gradient estimates from environment experience.

Explore the policy gradient theorem, defining policy performance as the value of the initial state, and show how the gradient links returns to state frequencies and action probabilities for improvement.

Discover the reinforce algorithm, a policy gradient method that uses Monte Carlo episode returns to update a neural policy via the policy gradient theorem.

Learn parallel learning for policy gradient methods to address similar states by running multiple environments in parallel, alongside experience replay for diverse updates in neural networks.

Apply entropy regularization to maintain high policy entropy in policy gradient learning, promoting exploration, robustness to unseen states, and finer optimization of action choices.

Initialize the policy neural network and parallel environments to collect trajectories across episodes. Backpropagate returns with gamma, apply entropy, and update the policy via gradient ascent with alpha.

Explore how PyTorch Lightning streamlines deep learning with the Lightning Module and trainer to automate training, validation, and testing, while leveraging callbacks and logging to run on multiple devices.

Implement a policy gradient method using a three-layer neural network with softmax outputs in a gym cartpole task, visualizing untrained action probabilities and setting up the environment.

Create a multi-environment setup for reinforce by using gym's vector environment to run two parallel copies, observe states, actions, rewards, and wrap with normalization for stable policy learning.

Create a dataset class to collect and structure batched transitions from a policy interacting with an environment, compute discounted returns, and prepare data for training a policy gradient algorithm.

Create the reinforce algorithm by building a class that extends a Lightning module. Configure hyperparameters, a policy network, a dataset, and an AdamW optimizer with a data loader.

Implement the second part of the REINFORCE algorithm by computing log probabilities, applying entropy regularization, and updating the policy network via a batch training step using PyTorch Lightning.

Observe policy improvement as entropy decreases during training, moving from exploration to exploitation to keep the bowl upright; tests transfer normalized observations and reveal action probabilities.

Explore extending policy gradient methods to continuous action spaces using the REINFORCE algorithm with a normal distribution, computing mean, std, log probabilities, and entropy for policy updates.

Implement a reinforcement learning gradient policy for continuous actions and test it in a Brax robotics environment. Use a normal distribution with mean bounded by tanh and std by softplus.

Create and configure an inverted pendulum environment in gym using the Barracks library, register and wrap observations and rewards, and test with a random policy before training a gradient policy.

Create a dataset tailored for continuous-action policy gradient training by collecting transitions, computing returns with gamma, stacking observations and actions with torch.stack, and yielding shuffled batches for reinforce.

Define the REINFORCE policy gradient in a lightning module, configuring environments, the neural policy, gamma, entropy, optimizer, and an RL dataset with train and test loaders.

Develop a continuous-action policy gradient algorithm by computing the mean and standard deviation from observations, forming a normal distribution, and optimizing policy loss with entropy regularization.

Learn to combine policy gradient with temporal difference in advantage actor critic, using neural network estimates of state values to bootstrap advantages and update the policy during episodes.

Implement the advantage actor critique algorithm by building a neural policy to model action distribution and a critic to estimate state values, with a 128-unit hidden layer.

Create and render a pendulum environment with gym for advantage actor critique algorithm, run parallel copies, inspect observations and actions, scale actions via tanh, and wrap with statistics and normalization.

Create an rl dataset class that uses a policy to generate action distribution parameters, sample actions, and yield experiences for training across steps per epoch, updating observations with no-grad annotation.

Implement the A2C algorithm by building the Advantage Vector Critique class, configuring dual optimizers for policy and value networks, and setting up environments and data loaders.

Implement the A2C training step by updating the value and target value networks using gamma and done flags, then update the policy with log probabilities, advantages, and entropy.

After reducing epochs from 3500 to 2000 to prevent overfitting, the agent balances the pendulum with policy gradient methods, improving returns from -1400 to -200 and demonstrating test episodes.