Reinforcement Learning beginner to master - AI in Python

Explore the five elements common to all control tasks—state, actions, rewards, agent, and environment—and build a general reinforcement learning template using chess, a robotic arm, and Pacman.

The Markov decision process is a discrete-time stochastic control template where the agent observes a state, acts, receives a reward, and transitions to a new state with memoryless dynamics.

Explore the types of Markov decision processes by distinguishing finite and infinite MDPs, and episodic versus continuing tasks, with examples like a 5x5 maze and driving a car.

Explore the difference between trajectory and episode in reinforcement learning: a trajectory is the state-action-reward sequence, while an episode starts at the initial state and ends at the final state.

Explain the discount factor gamma and its role in shaping returns by discounting future rewards, encouraging efficient, long-term planning in reinforcement learning.

Explore the agent's policy pi, a mapping from states to actions, and distinguish deterministic from stochastic policies; the goal is the optimal pi* that maximizes the discounted rewards.

Learn to evaluate states and actions with v(s) and q(s,a) using returns under policy pi, and see how q-values simplify the search for the optimal policy.

Discover how to solve a Markov decision process by maximizing the expected return through optimal state values and q-values, and derive the optimal policy via Bellman optimality equations.

Open the MDP_introduction notebook to explore how a reinforcement learning agent interacts with a 5x5 maze using gym, actions, rewards, and the Markov decision process.

Construct trajectories and episodes in a gym environment by recording state, action, reward, done, and next state; compute discounted returns with gamma and test a random policy.

Explore value iteration, a dynamic programming method to compute the optimal policy by iteratively updating state values until convergence, using a 5x5 maze example.

implement the policy iteration algorithm for a 5x5 maze environment using NumPy and PyPlot, visualize states and actions, and initialize with a random policy before converging to the optimal policy.

Define the agent's policy with a 5x5x4 probability table initialized to 0.25, and visualize the random policy's performance before applying value iteration to improve action decisions.

Initialize a 5x5 state value table with zeros, visualize it with plot_values against the environment frame, and prepare to implement value iteration for optimal values and policy.

Define a value iteration function with policy probabilities, state values, theta, and gamma, then use simulate_step within a model to update values and the policy.

Apply policy evaluation by computing state values as the expected, discounted return under the policy via the Bellman equation, updating estimates until convergence.

Apply policy iteration to a 5x5 maze by initializing the environment, a 5x5 policy of four-action vectors with 0.25, and a zero value table, then render and test random policy.

Implement policy evaluation as part of policy iteration by iterating until delta <= theta, updating the value table from action probabilities, rewards, and gamma-discounted returns.

Improve the current policy using the state value function and q-values to identify better actions, implement pi prime, and converge to the optimal policy via the policy improvement theorem.

Implement policy improvement in policy iteration by building policy_improvement with three inputs (policy probabilities, state values, gamma), evaluating actions via returns, and updating the policy and stability flag.

Integrate policy evaluation and policy improvement in a single policy_iteration function, tune with theta and gamma, and plot policy and value changes to converge on the optimal policy.

Explore generalised policy iteration (GPI) as the core template of reinforcement learning, alternating policy evaluation and improvement through environment interaction and trial and error.

Explore solving control tasks with Monte Carlo methods within generalized policy iteration, using q-values and discounted returns to evaluate and improve policies without a model.

Learn on-policy Monte Carlo control with an epsilon-greedy policy, updating q-values by averaging returns and using backwards return calculation across episodes to balance exploration and exploitation.

Implement an on-policy Monte Carlo control algorithm that learns from full episodes in a 5x5 maze, updating q-values from returns and refining an epsilon-greedy policy.

Implement on-policy Monte Carlo control to learn from experience by updating q-values as the average of observed returns for state-action pairs, using a policy and epsilon exploration across multiple episodes.

Demonstrates on-policy Monte Carlo control results with a q-value table and action-value plots, identifying optimal actions toward the goal, the learned policy, and exploration, then introduces off-policy Monte Carlo.

Explore off-policy learning with two policies, use exploratory and target policy, leverage importance sampling to update q-values, and achieve robust policy improvement.

Implement off-policy Monte Carlo in a 5x5 maze by training a target policy to pick the action from a q-value table while an exploratory policy collects experience with epsilon-greedy exploration.

Off-policy monte carlo control updates q-values from reversed episode returns using importance sampling with a csa table, gamma, and epsilon under target and exploratory policies.

Demonstrate off-policy Monte Carlo learning using a q-table where only states on the optimal path are refined, and use a single policy with exploration plus separate exploration and improvement policies.

Temporal difference methods learn from experience and combine Monte Carlo and dynamic programming ideas. They update Q-values to guide policy without a model, using bootstrapping and generalized policy iteration.

Apply temporal difference methods to solve control tasks by updating a state-action Q value table using TD error, rewards, and discounted next-state values under a policy.

Learn the on-policy sahsa method with epsilon-greedy exploration, updating the q-value table via the temporal-difference error using the sahsa five elements and an alpha step.

Learn to implement the sarsa algorithm by importing libraries, creating a 5x5 maze environment, initializing a zero q-value table, and using an epsilon-greedy policy to choose actions.

Learn to implement the SARSA algorithm with an epsilon-greedy policy, updating a Q-value table using alpha and gamma. Run episodes in an environment to derive an optimal policy.

Explore q-learning, an off-policy temporal-difference method with two policies: a greedy target Pi and exploratory B. The agent gathers experiences, updates Q-values, and reaches the optimal policy.

Master q-learning with off-policy exploration using a greedy target policy and a stochastic exploratory policy. Build and initialize the q-value table, and compare the target and exploratory policies.

Implement the q-learning algorithm with a q-value table, exploratory and target policies, updating via alpha and gamma across episodes, illustrating off-policy learning and testing with the target policy.

Explore n-step temporal difference methods, bridging Monte Carlo and temporal difference learning, using n-step bootstrapping and the SARSA update rule to combine actual rewards with estimates.

Explore how n-step methods connect Monte Carlo and temporal-difference methods in reinforcement learning, with SARSA as an n-step variant and n tuning toward one extreme or the other.

Apply n-step SARSA, an on-policy extension using n-step bootstrapping to update q-values with multi-step returns and epsilon-greedy action selection.

Explore n-step SARSA in action by collecting eight rewards to form the eight-step return and update Q-values after eight moves, showing how longer returns yield more reliable estimates.

Combine temporal-difference SARSA with n-step bootstrapping to update q-value table using n rewards and future estimates, with an on-policy epsilon-greedy policy guiding actions toward the 5x5 maze exit.

Learn to solve control tasks with continuous state spaces by transforming states into a finite representation using state aggregation and tile coding, as a first approach to enable finite-state methods.

Learn how to adapt tabular reinforcement learning methods to continuous state spaces using state aggregation and tile coding, including wrapping the environment and discretizing states.

Solve the mountain car task using the RSA algorithm with aggregated states, compare the original and 20x20 discretized state spaces, then build a Q-value table and an epsilon-greedy on-policy policy.

Learn tile coding to approximate the optimal value function in continuous state spaces by averaging estimates from multiple independent grids, reducing discretization error in tasks like golf and mountain car.

We implement tile coding to handle continuous states by creating multiple independent state aggregations and averaging their q-value estimates, improving accuracy in the mountain car task.

Combine tile coding with the RSA algorithm to solve the mountain car control task, creating four-fold state aggregations and a q-value table updated via mean across aggregations.

Explore function approximators to handle continuous state spaces, replacing tabular methods with parameterized models like linear and polynomial estimators, and learn how evaluation-improvement cycles converge to the optimal value function.

Explore artificial neural networks as flexible function approximators and learn the basics of feedforward networks for deep reinforcement learning, including input, hidden, and output layers and fully connected layers.

Explore how artificial neurons imitate biological cells, aggregating weighted inputs through synapses and applying activation functions like relu and sigmoid to produce outputs.

Minimize the cost function using stochastic gradient descent with environment rewards and neural network estimates. Backpropagation in PyTorch computes the gradient and updates the neural network parameters with alpha.

Learn to optimize a neural network to estimate Q values by minimizing mean squared error with environment samples. Understand targets from reward plus discounted next-state estimates and local/global minima.

Extend the temporal difference method to estimate the Q values with a neural network, producing deep sarsa that uses an epsilon-greedy policy and trains via stochastic gradient descent.

Optimize the deep q-network to produce increasingly accurate q-value estimates by minimizing the mean squared error between estimated and one-step target q-values using a batch of replay memory experiences.

Define a PyTorch-friendly environment wrapper for gym, converting states, rewards, and done signals to batched tensors, enabling seamless step and reset integration for deep sarsa training.

Visualize how a neural network estimates the cost to go, the negative of the highest Q value, for each state. Observe the state space map of car position and velocity.

Implement a replay memory to store state transitions (state, action, reward, next state) and sample batches to improve neural network estimates, using insert, sample, can sample, and Len.

Define can_sample to ensure memory has at least ten times batch size, then implement sample by drawing a batch and unpacking into states, actions, rewards, next states, converting to tensors.

Implement the deep SARSA training loop with a neural network for q-values, an epsilon-greedy policy, replay memory, and AdamW optimizer, updating over many episodes and tracking cost and returns.

Demonstrates evaluating a deep SARSA agent and visualizing learning progress with plot stats and cost to go. Interprets a neural-network based policy for a car environment.

Explore the deep q-learning algorithm, combining temporal difference q-learning with neural networks to estimate q-values from states. Employ off-policy, epsilon-greedy exploration, a target network, and replay memory for stable updates.