Artificial Intelligence 2.0: AI, Python, Deep RL + LLM Prize

Explore the twin delayed ddpg model, extending deep deterministic policy gradient to continuous action spaces, with twin critics, actor and critic targets, and an off-policy initialization using experience replay memory.

Learn how td3 q-learning works as an off-policy method with experience replay, explore actions via random sampling and gaussian noise, and train two critics with a minimum q-value target.

Explore the policy learning part of twin delayed ddpg, showing how the actor updates via gradient ascent using the critic's q-value and how polyak averaging and delayed updates stabilize learning.

Demonstrate the Td3 training process by computing targets from next states, updating two critics via MSE loss, performing gradient ascent on the actor, and syncing the actor and critic targets.

Begin from scratch in Google Colab, open a Python 3 notebook, and implement the 15 TD three reinforcement learning steps across cheetah, humanoid, and ant with interactive exercises.

Build two PyTorch neural networks—the actor model and actor target—with a shared architecture using 400 and 300 hidden neurons, relu and tanh activations, softmax scaling, and max action clipping.

Implement two critic neural networks and two critic targets with an architecture (400 hidden and 300 hidden) that take the concatenated state and action as input and output Q values.

Kick off the td3 training implementation by creating the actor, critic, and target networks, their optimizers, and a td3 class; choose cpu or gpu and prepare for sampling transitions.

Implement select action by formatting state as a torch tensor, feeding the actor to get a numpy action with noise and clipping, and outline the train method with memory batches.

Step four of the td3 training tutorial samples transitions from the replay buffer to create four batches—states, next states, actions, rewards and dones—and converts them to torch tensors.

Implement step five by feeding the next state batch into the actor target to obtain the next action prime, then prepare step six with Gaussian noise and clipping.

Generate a batch of noise with zero mean and policy noise sigma. Clip to [-noise clip, noise clip], add it to next actions, and clip to [-max action, max action].

implement step 9 by using the critic target to compute two Q values, q1 and q2, from the next state and next action, and take their minimum.

Compute the minimum of target Q1 and Q2 with torch, then form the final target as reward plus gamma times that minimum, adjusted by the done indicator.

Implement step ten by feeding the current state and action into two critic models to yield Q1 and Q2 and prepare their mean squared error loss against the target.

Implement critic loss as the sum of two mean squared error losses between each critic’s predictions and target, then backpropagate with an adam or sgd optimizer to update both critics.

Backpropagate the critic loss to update the twin critic models via stochastic gradient descent, after initializing gradients and computing them with backward, then perform gradient ascent on the actor.

Learn step 15 of this twin delayed ddpg tutorial, updating the actor every two iterations with gradient ascent on the first critic's q value to maximize expected return.

Apply Polyak averaging to update the actor target and critic target weights every two iterations, using parameter-wise updates through for loops and PyTorch parameters.

Implements td3 with twin critic targets updated by polyak averaging, in PyTorch, and evaluates the policy over ten episodes in pybullet environments (half cheetah, ant, half humanoid).

Explore step 18 of the Td3 implementation: save and load actor and critic weights, and evaluate the policy over ten episodes in pybullet gym environments.

Train and evaluate a TD3 agent over 500,000 time steps using replay memory and off-policy learning to improve average rewards across episodes. Start with 10,000 random actions before policy-driven exploration.

Explore inference in a td3 agent by loading trained weights, running ten evaluation episodes, and generating videos in a 3d aunt environment, with actor and critic networks.

Demonstrates training three agents (ant, half cheetah, half humanoid) with td3 on pybullet gym environments, training up to 500,000 to 1,000,000 timesteps and showing final inference videos.

Watch the td3 training progress from exploration to inference, noting time steps, average rewards, and final videos of an ant walking the field.

Explore neurons and activation functions, then examine how neural networks work with gradient descent and stochastic gradient descent, using a housing price example to illustrate backpropagation and running networks.

Learn how a neuron serves as the building block of artificial neural networks, computing a weighted sum of input values, applying an activation function, and passing signals via synapses.

Explore the activation function in deep learning, compare threshold, sigmoid, rectifier, and hyperbolic tangent, and apply them in hidden and output layers to shape neural network outputs.

Learn how neural networks work with a property valuation example, exploring input, hidden, and output layers, weights, and activation functions, and applying a pre-trained model to estimate price.

Reveal how neural networks learn by replacing hard-coded rules with a single-layer perceptron that adjusts weights to minimize the cost function through backpropagation.

Explore gradient descent, guided by backpropagation, as the efficient way to update neural network weights by minimizing the cost function with its slope, instead of brute-force searches.

Stochastic gradient descent avoids non-convex traps by updating weights after each training example, offering faster, lighter learning than batch gradient descent and introducing mini-batch variants for balance.

Explore how backpropagation simultaneously adjusts all neural network weights via forward and backward passes, detailing weight initialization, learning rate, and batch or epoch training.

Explore reinforcement learning concepts, including the Bellman equation and Q-learning, outline the plan for this section, and study Markov decision processes, policies versus plans, and temporal difference with a visualization.

Explain reinforcement learning, where an agent in an environment takes actions, changes state, and receives rewards to learn optimal behavior.

Unpack the Bellman equation in reinforcement learning, linking states, actions, rewards, and gamma to compute V(s) via the max over actions toward the finish.

Convert maze state values into a treasure map of arrows to guide the AI’s best moves toward the goal, using the plan and contrasting with future policy concepts.

Learn how Markov decision processes model decision making under randomness, contrasting deterministic and stochastic search, and extending the Bellman equation to expected values for planning.

Explore policy versus plan in a stochastic Markov decision process, showing how randomness and the Bellman equation reshape state values and drive learned policies over preplanned paths.

Explore how a living penalty affects the Bellman equation in reinforcement learning by adding a small reward of -0.04 per move, steering the agent toward faster finishes.

Explore q-learning by comparing action quality (q values) to state values, using the Bellman equation, rewards, and discounting within Markov decision processes to pick the best action.

Explore temporal difference in Q-learning, updating Q-values with rewards and future state values, while balancing stochastic environments and convergence through learning rate alpha.

Explore q-learning in a gridworld maze in the artificial intelligence 2.0 course, visualize q-values and learned policy, and see how exploration, randomness, and discounting shape reinforcement learning outcomes.

Plan of attack for deep q-learning covers the learning and acting components, neural network weight updates, temporal difference extension to deep q-learning, experience replay, and exploration versus exploitation policies.

Explore how deep q-learning extends q-learning by feeding state vectors into a neural network to produce action q-values, compare predictions to targets, and update weights through backpropagation.

Explains deep q-learning from learning to acting: using fixed q-values, applying softmax for action selection, and updating a neural network through backpropagation across epochs from state vectors.

Experience replay bolsters deep q-learning by storing experiences in memory, sampling a uniform batch to break sequential bias, and enabling learning from rare, diverse states.

Learn action selection policies for deep q-learning, including epsilon greedy, epsilon soft, and softmax, to balance exploration and exploitation based on q-values.