Curiosity Driven Deep Reinforcement Learning

Review reinforcement learning basics, including agent–environment interactions, rewards, and discounting with gamma. Explain actor-critic methods using a critic for value and an actor for policy.

Explore a basic actor critic implementation in PyTorch, detailing a policy and value network, action selection with a categorical distribution, and temporal difference learning updates.

Explore asynchronous advantage actor-critic methods, parallel learning with a global agent, and use generalized advantage estimation, entropy regularization, and a shared optimizer to enable on-policy learning without replay memory constraints.

Outline the modular structure of this deep reinforcement learning project, detailing the main file, parallel environment file, worker file, actor critic file, utils, and open ai gym integration, with a shared adam optimizer and memory handling.

Develop a practical reading strategy for deep learning papers: skim for the core idea, implement a quick prototype, then study experiments, results, and architecture to translate ideas into code.

Analyze the A3C abstract and introduction, detailing a synchronous actor-critic method that runs on CPUs, beats Atari state-of-the-art results, and uses multiple independent agents with uncorrelated experiences for on-policy learning.

Learn a practical crash course in parallel processing with Python, using the PyTorch multithreading package and multiprocessing, setting environment threads, and implementing spawn-based workers to run parallel episodes.

Explore the A3C asynchronous reinforcement learning framework with distributed actors and learners, central gradient updates, and global experiences; review value-based and policy-based methods, q-learning, end-step returns, and actor-critic basics.

Explore the asynchronous reinforcement learning framework and A3C, detailing network architecture and multi-threaded training. See how no replay memory, entropy regularization, and parameter space exploration enable efficient learning.

Develop and code an asynchronous reinforcement learning actor-critic network with a deeper cnn and a gated recurrent unit, producing a softmax policy and value output, for 4x42x42 inputs.

Learn to overcome credit assignment bias in policy learning by applying generalized advantage estimation within an actor-critic framework, using gamma, lambda, and entropy-regularized losses.

Implement a minimalist replay memory by creating a memory class with lists for the value function, log probs, and rewards, with add, clear, and retrieve methods to store recent transitions.

Implement a shared Adam optimizer for reinforcement learning by deriving from Adam, iterating over parameter groups to initialize state variables, and sharing exponential moving averages and their squares in torch.

Explore the experiments and discussion of asynchronous A3C variants across Atari, Talks 3D, Djoko, and Labyrinth, comparing performance to human starts and prioritized experience replay in deep learning.

Explore modifying openai gym Atari environments with wrappers for image pre-processing and frame stacking. Convert to grayscale, resize to 42×42, repeat actions four times, and normalize to 0–1.

Implement the asynchronous advantage actor-critic loop for curiosity-driven deep reinforcement learning, wiring the main loop with parallel environments and shared memory.

Examine how curiosity-driven deep reinforcement learning uses intrinsic rewards from predicting state transitions in a reduced feature space to learn with sparse or no extrinsic rewards, enabling transfer across environments.

Curiosity drives learning in sparse-reward environments by rewarding prediction errors of a forward dynamics model, using inverse dynamics features to predict actions, tested in Doom Wisdom and Mario.

Explore how the intrinsic curiosity module combines an inverse and forward model with a feature encoder to produce intrinsic rewards, guiding exploration alongside extrinsic rewards in a generalized framework.

Implement the intrinsic curiosity module (icm) within a deep reinforcement learning agent, detailing CNN-based networks, inverse and forward models, hyperparameters, and the experiment setup in gym many world environments.

The lecture examines ICM experiments across dense, sparse, and very sparse rewards in Doom and Mario, highlighting robust exploration without extrinsic rewards and advantage over pixel-based ICM.

Set up the many world environment and train an ICM agent to navigate a 3D hallway, reach a red box for extrinsic reward, and observe intrinsic rewards shaping exploration.

Set up a Python 3.8 virtual environment, install torch 1.13 and compatible gym/gymnasium versions, and fix reset/step usage to handle observation, reward, done, truncated, and info.

Align the agent pipeline with the new gym interface by fixing observation shapes, reset handling, and terminal or truncated flags, and switch to gymnasium for mini world environments.