Reinforcement Learning Masterclass

Explore core statistical symbols and set theory concepts like mu, sigma, variance, covariance, correlation rho, arg max and arg min, set operations, and real spaces R and R^n.

Explore probability basics—from sample space and events to independence and conditional probability—then apply Bayes' rule, Bayesian inference, and Naive Bayes to uncertainty in machine learning.

Explore Markov decision processes to make optimal decisions under uncertainty by defining states, actions, transitions, and rewards, and by using value and policy iteration to maximize total reward over time.

Model a dynamic pricing scenario as a Markov decision process to maximize discounted revenue across periods by selecting price actions based on current demand states, considering transition probabilities and rewards.

Apply backward induction in a Markov decision process to optimize revenue across states and actions, using transition probabilities, rewards, gamma, and initialized value and policy matrices.

Explore how a policy matrix in a Markov decision process maps time periods and demand states to optimal price levels to maximize revenue.

Dynamic programming decomposes complex problems into subproblems, uses memoization and the Bellman equation, guided by the optimality principle, to build optimal policies across stages and states.

Logistics companies maximize fleet value by solving a multi-dimensional knapsack problem with trucks constrained by weight, volume, capacity, and cargo type, using a dynamic programming approach for real-time decisions.

Analyze the dynamic programming model for a knapsack problem, defining items and constraints with binary decision variables x_I to maximize total value under weight and volume capacities.

Explore dynamic programming for a multidimensional knapsack, using item values and weights across dimensions to maximize value within given capacities via a dp table.

Analyzes a multidimensional knapsack with dynamic programming to show how the algorithm updates the best combination to reach a maximum value of 50 within weight and volume capacities.

Explore discrete and continuous probability distributions, including binomial, Poisson, geometric, normal, exponential, gamma, beta, and Weibull, with real-world applications in quality control, reliability, and forecasting.

Explore how the Bellman equation links state values to future rewards through dynamic programming, optimal substructure, and value functions in Markov decision processes, including value iteration, policy iteration, and Q-learning.

Explore policy approximations for scalable, data-driven decision making in complex environments, from value-based and policy gradient methods to advanced actor-critic architectures and real-world RL applications.

Explore the universal five-element framework for sequential decisions and compare policy classes—PFA, CFA, VFA, DLA—covering policy search and lookahead, with hybrids like actor-critic and Monte Carlo tree search.

Explore dynamic programming fundamentals—memoization, overlapping subproblems, and optimal substructure—and see how top-down and bottom-up approaches optimize problems like Fibonacci and knapsack.

Compute the state value function v_pi(s) as the expected gamma-discounted sum of rewards under a policy, via the Bellman expectation equation and iterative evaluation for convergence.

Explore the iterative policy evaluation algorithm to compute the state value function for each state under a policy, using transition probabilities, rewards, gamma, and theta in a three-state example.

apply monte carlo methods for rl to learn from full episodes, update visited states using actual returns, compare first-visit and every-visit approaches, and improve policies with epsilon-greedy exploration.

Explore blackjack as a reinforcement learning problem where an agent learns to hit or stand using total, dealer's visible card, and a usable ace, with q-learning to develop a policy.

Learn to play blackjack with a Python reinforcement learning agent using a Q-table and epsilon-greedy policy. Train and evaluate in a blackjack environment with shaped rewards and dynamic learning rates.

Analyze reinforcement learning outputs from blackjack training, track win, draw, loss rates and epsilon decay across 200,000 episodes to reveal a near-optimal strategy.

Explore the sarsa algorithm, its on-policy updates, and how it differs from q-learning using td errors, alpha learning rate, gamma discount, epsilon-greedy exploration, and cliff walking intuition.

Implement SARSA for taxi v3 in Python using gymnasium, numpy, and matplotlib; initialize q-table, epsilon-greedy policy, and update q-values through SARSA training across episodes.

Explore implementing SARSA in the taxi environment by building frame-by-frame animation, initializing the gym taxi v3, capturing frames to GIF, and using a Q-table to navigate pickups and drop-offs.

Master Q-learning, a model-free, off-policy method that estimates the best action values, updates toward reward plus discounted max future value using gamma, with epsilon-greedy exploration and alpha tuning.

Explore the frozen lake environment from start to goal using q-learning, with epsilon-greedy exploration, updating q-values across episodes to learn an optimal path while avoiding holes.

Explore how to implement q-learning in a frozen lake environment with Python, building a q-table, epsilon-greedy exploration, and evaluating learning through training and testing.

Explore a Python cliff-walking grid-world, where an agent learns to reach the bottom-right goal while avoiding a cliff, using Q-learning with epsilon-greedy exploration and a learned policy.

Explore function approximation in reinforcement learning, replacing lookup tables with a generalizing function and training it with gradient descent across linear, polynomial, radial basis, and neural networks.

Explore how neural networks approximate value functions and policies in reinforcement learning, using deep q networks, actor-critic, and PPO for stable updates. Applications include robotics, autonomous vehicles, and resource management.

Tile coding discretizes a continuous state space with three 4x4 tilings, creating 48 sparse linear features for value function approximation and enabling generalization via offset tilings.

Learn how reinforce, a policy gradient algorithm, learns a policy mapping states to action probabilities via full episodes and returns. Update uses gradient ascent with optional baselines to reduce variance.

Implement an actor-critic reinforce with baseline in python, using a shared network with policy and value heads, softmax action probabilities, entropy regularization, and training on Cartpole.

Balance bias and variance in policy gradient updates with generalized advantage estimation (GAE) by weighting TD errors with gamma and lambda, guiding trajectory collection and policy and value updates.

Explain generalized advantage estimation (GAE) and its use in PPO to reduce variance while preserving learning signals, via TD errors, gamma, and lambda, and normalization of advantages across batches.

Explore the advantage actor critic (A2C) method, a synchronized actor-critic reinforcement learning algorithm that uses a critic to stabilize updates and guide the actor toward better actions.

Combine asynchronous advantage actor-critic (a3c) with parallel training by running multiple workers in separate environments to update a shared global network, accelerating learning on CPUs.

Explore deterministic policy gradient for continuous action spaces, featuring the actor-critic architecture, target networks, experience replay, and the advantages over stochastic policy gradients.

Explore deep deterministic policy gradients, an actor-critic, model-free method for continuous action spaces that uses replay buffers and target networks for stable learning.

TD3 strengthens deterministic policy gradient with twin critics, delayed actor updates, and target policy smoothing for stable learning in continuous control tasks, aided by a replay buffer and target updates.

Learn how soft actor-critic (SAC) combines off-policy learning with high policy entropy to balance exploration and performance, using a gaussian policy, dual critics, and an adaptive entropy coefficient alpha.

Understand trust region policy optimization and how KL divergence constrains policy updates for stability. Learn practical steps like surrogate objective, conjugate gradient, line search, and key hyperparameters delta, gamma, lambda.

Master trust region policy optimization (TRPO) to stabilize policy updates in continuous or discrete action spaces by using a policy and value network, collecting experiences, calculating advantages, and updating policy.

Implement discounted returns and advantages to train policies with TRPO, including surrogate and value losses, KL divergence constraints, and conjugate gradient optimization for stable updates.

Implement backtracking line search in TRPO to find a safe, effective policy update within a KL divergence constraint, using surrogate loss, conjugate gradient, and step-size tuning.

train a TRPO agent on CartPole v1 with OpenAI gym and PyTorch, collecting trajectories, updating the value function, and refining the policy across epochs; evaluate and save the best policy.

TRPO refines its policy in a CartPole trial, lifting mean rewards toward about 1000, with best evaluation near 10.11 and final test average around 9.15.

PPO uses a clipped surrogate objective and first order optimization to deliver stable, sample efficient policy updates, reusing data across epochs while balancing performance and simplicity.

Explore metal TRPO, a model ensemble approach to trust region policy optimization using synthetic rollouts. It blends multiple models, uncertainty handling, and a KL constraint for stable, efficient learning.

Explain how deep q-networks replace the q-table with a neural network that outputs q-values. Describe the q-network and target network, plus replay buffers and minibatch training to stabilize learning.

Explore hierarchical reinforcement learning, combining high, mid, and low level policies to set subgoals, use intrinsic rewards, and reuse skills through options, feudal networks, and manager and worker architectures.

Explore hierarchical reinforcement learning with options, a class managing temporally extended actions, learning via q-values, epsilon-greedy exploration, and a training loop for agents that select and execute options.

Learn hierarchical reinforcement learning with the four rooms gym environment, training a hierarchical agent, and track progress with rewards, episode length, and per-option plots.

HRL Python outputs show a learning curve where average rewards move from negative to positive by episode 80, as a hierarchical agent navigates subgoals with improving options.