Advanced Reinforcement Learning: policy gradient methods

Explore finite and infinite Markov decision processes, and episodic versus continuing tasks, with examples like a 5x5 maze and a car's continuous states.

Explore how the discount factor gamma shapes reward planning in reinforcement learning, balancing immediate vs long-term gains by discounted returns in a maze task.

Uncover the Bellman equations for state value and Q-value, expressing expected returns as discounted rewards from successor states, enabling policy evaluation and solving control tasks in reinforcement learning.

Explore Monte Carlo methods by learning from experience to estimate state-action values under a policy, using returns to update estimates and rely on law of large numbers without a model.

Implement on-policy Monte Carlo control with an epsilon-greedy policy that occasionally selects random actions; update action-value estimates by averaging returns across episodes to derive a near-optimal policy.

Temporal difference methods learn from experience to update Q-values and policy during an episode, blending Monte Carlo and dynamic programming with bootstrapping and generalized policy iteration.

Explore the family of n-step temporal-difference methods that blend Monte Carlo and TD learning, using one to multiple actual rewards and bootstrapped estimates to form step return targets.

Explore how function approximators replace tabular value estimates with parameterized models for continuous state spaces. Compare linear and polynomial estimators, learning weights to achieve memory-efficient, adaptable value approximation.

Explore artificial neurons that aggregate weighted inputs, apply activation functions like ReLU and sigmoid, and propagate signals through hidden and output layers.

Learn how stochastic gradient descent minimizes the neural network cost by using environment rewards, computing the gradient via backpropagation, and updating parameters to move toward minima.

Learn policy gradient methods, where a neural network defines action probabilities, producing stochastic policies that offer smoother learning and address limits of value based approaches.

Explore the policy gradient theorem, defining policy performance as the value of the initial state, and show how the gradient links returns to state frequencies and action probabilities for improvement.

Learn parallel learning for policy gradient methods to address similar states by running multiple environments in parallel, alongside experience replay for diverse updates in neural networks.

Initialize the policy neural network and parallel environments to collect trajectories across episodes. Backpropagate returns with gamma, apply entropy, and update the policy via gradient ascent with alpha.

Create a dataset class to collect and structure batched transitions from a policy interacting with an environment, compute discounted returns, and prepare data for training a policy gradient algorithm.

Implement the second part of the REINFORCE algorithm by computing log probabilities, applying entropy regularization, and updating the policy network via a batch training step using PyTorch Lightning.

Explore extending policy gradient methods to continuous action spaces using the REINFORCE algorithm with a normal distribution, computing mean, std, log probabilities, and entropy for policy updates.

Define the REINFORCE policy gradient in a lightning module, configuring environments, the neural policy, gamma, entropy, optimizer, and an RL dataset with train and test loaders.

Develop a continuous-action policy gradient algorithm by computing the mean and standard deviation from observations, forming a normal distribution, and optimizing policy loss with entropy regularization.

Create and render a pendulum environment with gym for advantage actor critique algorithm, run parallel copies, inspect observations and actions, scale actions via tanh, and wrap with statistics and normalization.

Implement the A2C algorithm by building the Advantage Vector Critique class, configuring dual optimizers for policy and value networks, and setting up environments and data loaders.