Deep Reinforcement Learning Projects with Python & PyTorch

learn to create and activate a conda virtual environment, install essential packages like numpy, pandas, matplotlib, gym, retro, opencv, pygame, and pytorch, and start a jupyter notebook.

Install the gem package with pip install gem. Load an environment with gem.make and inspect the observation space and action space to identify state shapes and discrete or continuous actions, then sample an action, call step, and render.

Download the resources folder and run a Jupyter Notebook from the desktop in a virtual environment to access Gridworld, Mountain Car, Ms. Pac-Man, and stock trading deep q-learning.

Explore a grid world environment where a robot navigates toward a gold medal to earn +10, avoids a hall with -10, and incurs -1 in other states.

Build a custom grid world environment with gym, initialize a 4x4 grid, and perform four discrete actions (left, right, up, down) while observing, rendering, and stepping.

Build a neural network with three layers to process gridworld state as a 3d array. Flatten to vector, apply relu activations, and output q-values for up, down, left, right actions.

Build a q-network in PyTorch by defining a neural network class with three linear layers, relu activation, and a forward pass mapping a 3d grid state to four action values.

Learn to train a neural network for reinforcement learning with MSE loss in PyTorch, using rewards and targets from the optimal Q function, updating weights via gradients.

explore how the agent balances exploration and exploitation with epsilon-greedy actions, starting with random actions to discover states, then gradually reducing exploration to rely on the neural network for decisions.

Train a neural network over thousands of episodes, preprocessing states, adding noise, and using an epsilon-greedy policy in PyTorch, then compute targets with a bellman equation and a target maker.

Train a neural network to estimate q-values for actions using PyTorch, with mean square error loss, Adam optimizer, and backpropagation, while decaying epsilon to balance exploration and exploitation.

Improve neural network training with batch training on many experiences to average gradients and update weights, then test the gridworld approach with random item placement against the single-sample method.

Train a neural network on batches with a replay buffer of experiences, memory 5000 and batch size 200, leveraging GPU acceleration for faster reinforcement learning.

Develop a reward metric that computes the mean of the last n rewards to track agent improvement during training and visualize mean rewards across about 10,000 episodes.

Explore why training instability occurs when targets drift and learn how a target network stabilizes learning by copying the agent's weights and updating every 50 steps.

build a target network by deep-copying the agent network to stabilize training, use it to generate next-state targets, and update its weights at a synchronize frequency of steps or episodes.

Import the gym package and load mountain car version zero environment, inspect observation and action spaces, then reset, step, and render to observe position, velocity, and actions left, no-op, right.

Explore how a dynamics neural network predicts the next state in the mountain car to build momentum, addressing sparse rewards and epsilon-greedy challenges with model-based exploration.

Explains implementing epsilon-greedy in a three-action mountain car environment with a neural network, including state shaping and episode-based rewards, and shows its ineffectiveness in sparse reward settings.

Build a dynamic neural network in Python with input as the current state plus a three-element one-hot action vector, outputting the two-element next state.

Use a multivariate Gaussian with position and velocity, a covariance matrix and mean, to estimate next states and pick action with the lowest probability to explore unseen or winning states.

Build a multivariate gaussian distribution from previous states using torch, compute mean and covariance, evaluate next states via log probabilities, and select the action with the lowest probability.

Demonstrates model-based exploration for mountain car, training a q-network and a dynamics network with a 0.002 learning rate and 50 past states for multivariate gaussian distribution during 10,000 exploration steps.

Explore how the ICM module reshapes rewards to promote visiting new or rarely seen states, and introduces filter, inverse, and forward neural networks, with each component covered in separate videos.

Describe the filter neural network, the first component of the ICM module, which removes noise from any input state to create a configurable vector representation.

Build a filter net in PyTorch, with three linear layers and relu activations followed by tanh, mapping the state input to a 3-node output on CUDA.

Explore how an inverse neural network acts as a classifier by combining current and next state representations from a filter network, using softmax to output action probabilities.

Build an inverse neural network in PyTorch by defining an inverse net class with three linear layers, ReLU activations, and a softmax classifier for batch action probabilities.

The forward network takes the current state from the filter network to predict the next state, updates weights via the loss, and assigns rewards based on prediction accuracy.

Build a forward neural network to predict next state representation in deep reinforcement learning. Use three linear layers with 32 nodes, ReLU, and tanh after one-hot action encoding and concatenation.

Build an agent Q network and a target Q network for the mountain car environment, using input shape two and output shape three, with three linear layers and ReLU activations.

Learn to train a Q network with ICM by processing batches, computing Bellman targets with a target network, and updating with ICM rewards and losses.

Train a Q-network with an intrinsic curiosity module (ICM) using a replay buffer, epsilon-greedy exploration, and a joint forward and inverse network loss to improve learning and rewards.

Explore the random network distillation (RND) module to handle sparse rewards and noise in deep reinforcement learning, using a fixed target network and a predictor to generate curiosity-driven intrinsic rewards.

Build pnet and tnet networks in python. Define a class with four linear layers, relu activations, and forward method that maps a two-shape state to a three-node output on GPU.

Build and train a q-network with an rnd module by constructing a replay buffer, epsilon-greedy exploration, and normalization to stabilize learning and improve mean rewards.

Develop reinforcement learning for a Flappy Bird game, using a convolutional network to process image states and apply rewards of +10 for passes, -1 for hits, and 0.1 otherwise.

Develop a flappy bird game in Python by importing the Gamestate class, using frame stepping with up and do-nothing actions, and preprocessing frames to 84×84 grayscale for faster network training.

Build a convolutional network for deep reinforcement learning by defining three convolutional layers and two linear layers with ReLU activations, producing an output equal to the action count.

Build a deep reinforcement learning training loop for a conv net q-network with epsilon-greedy exploration, replay memory, and target network updates using 84x84 grayscale frames stacked to four frames.

Move from a one-step to a two-step Q network by using two future rewards and the target network, computing a two-step target with R1, R2, gamma, and done flags.

Learn to implement a two-step convolutional q-network that processes stacked states, uses epsilon-greedy actions and a target network, and trains with two-step targets from a replay buffer, outperforming a basic q network.

Explore prioritized experience replay buffer to bias sampling toward experiences with higher losses, adjust with alpha and beta to balance exploration and bias correction, accelerating policy optimization.

Build a prioritized replay buffer with alpha 0.6 and beta starting at 0.4, updating priorities by losses and annealing beta to 1, improving mean rewards.

The dueling Q network splits into value and advantage streams, computes q values as v(s) plus a(s,a) minus the mean, and speeds training for Atari games.

In Python, implement a dueling q-network with a convolutional backbone and two branches for state value and advantages, combine to compute q-values; show faster training than a basic q-network.

Navigate Pac-Man with walls, paths, and four ghosts; eat dots and corner pellets, with -0.1 for hitting a ghost when lives remain, and -1 for lives, plus +1 during fear.

Create a Ms Pac-Man environment in Python using retro, converting frames to grayscale 84x84 and mapping eight actions. Render and skip nonessential states using score, timer, and lives.

Train a Q network to play Pac-Man using a conv net with eight actions, a replay buffer, epsilon-greedy exploration, a target network, and huber loss.

Explore n-step q networks, extending from one- and two-step to three steps and beyond. Compute multi-step targets by accumulating rewards and discounting with gamma using the target network.

Learn to implement an n-steps q-network in Python, featuring get batch and batch target functions, replay buffer sampling, multi-step rewards, and training with a target network.

Discover how noisy networks replace epsilon-greedy exploration with trainable noise in weights, using a noisy layer that adds Gaussian mu and sigma with non-trainable epsilon, enabling adaptive exploration.

Learn to implement a noisy network for deep reinforcement learning by replacing weights and biases with trainable Gaussian noise in a dueling Q network.

Build a noisy double dueling q-network with agent and target networks, and use agent-driven target selection to reduce maximization bias while training efficiently.

Build an agent that learns when to buy and sell shares using candlestick charts, open-high-low-close data, and basic indicators to predict market direction across time frames.

Process stock data with pandas by loading a CSV, renaming columns, and dropping unused fields. Filter candlesticks with epsilon checks and normalize high, low, and close relative to open prices.

Build a from-scratch trading environment for deep reinforcement learning by implementing a Yandex environment with initialization and step, state construction, actions, and reward logic.

Develop a dueling conv1d q network by building a convolutional feature extractor with two streams for state value and advantage, then combine to produce q values for actions.

Train a trading robot using deep reinforcement learning, building a replay buffer for a dqn with a target network and epsilon-greedy exploration.