Reinforcement Learning beginner to master - AI in Python

Explore the five elements common to all control tasks—state, actions, rewards, agent, and environment—and build a general reinforcement learning template using chess, a robotic arm, and Pacman.

Explore the difference between trajectory and episode in reinforcement learning: a trajectory is the state-action-reward sequence, while an episode starts at the initial state and ends at the final state.

Explain the discount factor gamma and its role in shaping returns by discounting future rewards, encouraging efficient, long-term planning in reinforcement learning.

Explore the agent's policy pi, a mapping from states to actions, and distinguish deterministic from stochastic policies; the goal is the optimal pi* that maximizes the discounted rewards.

Learn to evaluate states and actions with v(s) and q(s,a) using returns under policy pi, and see how q-values simplify the search for the optimal policy.

Discover how to solve a Markov decision process by maximizing the expected return through optimal state values and q-values, and derive the optimal policy via Bellman optimality equations.

Open the MDP_introduction notebook to explore how a reinforcement learning agent interacts with a 5x5 maze using gym, actions, rewards, and the Markov decision process.

Explore value iteration, a dynamic programming method to compute the optimal policy by iteratively updating state values until convergence, using a 5x5 maze example.

implement the policy iteration algorithm for a 5x5 maze environment using NumPy and PyPlot, visualize states and actions, and initialize with a random policy before converging to the optimal policy.

Initialize a 5x5 state value table with zeros, visualize it with plot_values against the environment frame, and prepare to implement value iteration for optimal values and policy.

Apply policy iteration to a 5x5 maze by initializing the environment, a 5x5 policy of four-action vectors with 0.25, and a zero value table, then render and test random policy.

Implement policy evaluation as part of policy iteration by iterating until delta <= theta, updating the value table from action probabilities, rewards, and gamma-discounted returns.

Improve the current policy using the state value function and q-values to identify better actions, implement pi prime, and converge to the optimal policy via the policy improvement theorem.

Implement policy improvement in policy iteration by building policy_improvement with three inputs (policy probabilities, state values, gamma), evaluating actions via returns, and updating the policy and stability flag.

Integrate policy evaluation and policy improvement in a single policy_iteration function, tune with theta and gamma, and plot policy and value changes to converge on the optimal policy.

Learn on-policy Monte Carlo control with an epsilon-greedy policy, updating q-values by averaging returns and using backwards return calculation across episodes to balance exploration and exploitation.

Implement on-policy Monte Carlo control to learn from experience by updating q-values as the average of observed returns for state-action pairs, using a policy and epsilon exploration across multiple episodes.

Demonstrates on-policy Monte Carlo control results with a q-value table and action-value plots, identifying optimal actions toward the goal, the learned policy, and exploration, then introduces off-policy Monte Carlo.

Implement off-policy Monte Carlo in a 5x5 maze by training a target policy to pick the action from a q-value table while an exploratory policy collects experience with epsilon-greedy exploration.

Demonstrate off-policy Monte Carlo learning using a q-table where only states on the optimal path are refined, and use a single policy with exploration plus separate exploration and improvement policies.

Temporal difference methods learn from experience and combine Monte Carlo and dynamic programming ideas. They update Q-values to guide policy without a model, using bootstrapping and generalized policy iteration.

Learn the on-policy sahsa method with epsilon-greedy exploration, updating the q-value table via the temporal-difference error using the sahsa five elements and an alpha step.

Learn to implement the sarsa algorithm by importing libraries, creating a 5x5 maze environment, initializing a zero q-value table, and using an epsilon-greedy policy to choose actions.

Explore q-learning, an off-policy temporal-difference method with two policies: a greedy target Pi and exploratory B. The agent gathers experiences, updates Q-values, and reaches the optimal policy.

Master q-learning with off-policy exploration using a greedy target policy and a stochastic exploratory policy. Build and initialize the q-value table, and compare the target and exploratory policies.

Implement the q-learning algorithm with a q-value table, exploratory and target policies, updating via alpha and gamma across episodes, illustrating off-policy learning and testing with the target policy.

Explore n-step temporal difference methods, bridging Monte Carlo and temporal difference learning, using n-step bootstrapping and the SARSA update rule to combine actual rewards with estimates.

Explore how n-step methods connect Monte Carlo and temporal-difference methods in reinforcement learning, with SARSA as an n-step variant and n tuning toward one extreme or the other.

Apply n-step SARSA, an on-policy extension using n-step bootstrapping to update q-values with multi-step returns and epsilon-greedy action selection.

Combine temporal-difference SARSA with n-step bootstrapping to update q-value table using n rewards and future estimates, with an on-policy epsilon-greedy policy guiding actions toward the 5x5 maze exit.

Learn to solve control tasks with continuous state spaces by transforming states into a finite representation using state aggregation and tile coding, as a first approach to enable finite-state methods.

Learn how to adapt tabular reinforcement learning methods to continuous state spaces using state aggregation and tile coding, including wrapping the environment and discretizing states.

Learn tile coding to approximate the optimal value function in continuous state spaces by averaging estimates from multiple independent grids, reducing discretization error in tasks like golf and mountain car.

Explore function approximators to handle continuous state spaces, replacing tabular methods with parameterized models like linear and polynomial estimators, and learn how evaluation-improvement cycles converge to the optimal value function.

Learn to optimize a neural network to estimate Q values by minimizing mean squared error with environment samples. Understand targets from reward plus discounted next-state estimates and local/global minima.

Extend the temporal difference method to estimate the Q values with a neural network, producing deep sarsa that uses an epsilon-greedy policy and trains via stochastic gradient descent.

Optimize the deep q-network to produce increasingly accurate q-value estimates by minimizing the mean squared error between estimated and one-step target q-values using a batch of replay memory experiences.

Implement a replay memory to store state transitions (state, action, reward, next state) and sample batches to improve neural network estimates, using insert, sample, can sample, and Len.

Implement the deep SARSA training loop with a neural network for q-values, an epsilon-greedy policy, replay memory, and AdamW optimizer, updating over many episodes and tracking cost and returns.

Demonstrates evaluating a deep SARSA agent and visualizing learning progress with plot stats and cost to go. Interprets a neural-network based policy for a car environment.

Explore the deep q-learning algorithm, combining temporal difference q-learning with neural networks to estimate q-values from states. Employ off-policy, epsilon-greedy exploration, a target network, and replay memory for stable updates.