Mastering GPU Parallel Programming with CUDA: ( HW & SW )

Name: Mastering GPU Parallel Programming with CUDA: ( HW & SW )
Rating: 4.4 (673 reviews)

Performance Optimization and Analysis for High-Performance Computing

Bestseller

Created byHamdy egy

Last updated 10/2025

English

German [Auto],English [Auto],

What you'll learn

Comprehensive Understanding of GPU vs CPU Architecture
learn the history of graphical processing unit (GPU) until the most recent products
Understand the internal structure of GPU
Understand the different types of memories and how they affect the performance
Understand the most recent technologies in the GPU internal components
Understand the basics of the CUDA programming on GPU
Start programming GPU using both CUDA on Both windows and linux
understand the most efficient ways for parallelization
Profiling and Performance Tuning
Leveraging Shared Memory

Course content

12 sections • 58 lectures • 23h 3m total length

GPU vs CPU (very important)20:49
Explore the differences between CPUs and GPUs, focusing on Nvidia designs and the A100 streaming multiprocessor, memory hierarchy, and how parallel cores enable independent instruction processing in GPU workloads.
NVidia's history (How Nvidia started dominating the GPU sector)5:18
Explore Nvidia's rise from the NV1 device to the GeForce generation, highlighting memory, cores, and market dominance in GPUs and gaming.
Architectures and Generations relationship [Hopper, Ampere, GeForce and Tesla]16:07
How to know the Architecture and Generation6:56
discover how to determine a gpu’s generation and architecture using examples like the A100 and RTX 3090, including the GeForce and Tesla generations and the Ampere architecture.
The difference between the GPU and the GPU Chip4:50
Understand the difference between the gpu chip and the gpu itself. The chip is the silicon heart, while the gpu adds memory, cooling, and interfaces for consumer use.
The architectures and the corresponding chips5:25
Nvidia GPU architectures From Fermi to hopper12:24
Parameters required to compare between different Architectures24:05
Please don't skip this video. It is pivotal for the the whole course.
Half, single and double precision operations6:26
Compute capability and utilizations of the GPUs8:30
Explore how compute capability versions (major.minor) classify Nvidia GPUs, shaping feature support, tensor cores, and CUDA toolkit compatibility across architectures like Volta, Ampere, and Hopper.
Before reading any whitepapers !! look at this8:42
Explore NVIDIA GPU white papers as in-depth specifications to compare architectures. Learn to navigate the streaming multiprocessor design and track performance benchmarks and technical specs.
Volta+Ampere+Pascal+SIMD (Don't skip)52:08

What features installed with the CUDA toolkit?6:38
Installing CUDA on Windows4:48
Install Visual Studio as editor and compiler, install CUDA toolkit 12.3 on Windows with express installation, ensure 20–21 GB free space, and update Nvidia drivers to begin CUDA programming.
Installing WSL to use Linux on windows OS.6:26
Install Windows Subsystem for Linux to run Linux on Windows, choose a distribution like Ubuntu or Debian, and learn the wsl --install workflow to set up Linux for CUDA development.
Installing Cuda toolkits on Linux3:46
Install the Nvidia CUDA toolkit on Linux via WSL, verify Nvcc, run sudo apt install Nvidia CUDA toolkit, and confirm Nvcc shows release 11.5; note potential issues and future updates.

The course github repo
Mapping SW from CUDA to HW + introducing CUDA.12:49
Explore CUDA architecture and the host-device model, mapping blocks and warps to the four-level GPU hierarchy for efficient GPGPU on NVIDIA GPUs.
001 Hello World program (threads - Blocks)20:39
Compiling Cuda on Linux9:54
002 Hello World program ( Warp_IDs)9:03
003 : Vector addition + the Steps for any CUDA project22:30
004 : Vector addition + blocks and thread indexing + GPU performance18:21
#include <stdio.h>
#include <cuda_runtime.h>
#include <device_launch_parameters.h>

#define SIZE 2048 // Define the size of the vectors

// CUDA Kernel for vector addition
__global__ void vectorAdd(int* A, int* B, int* C, int n) {
int i = threadIdx.x + blockIdx.x * blockDim.x ;
C[i] = A[i] + B[i];
}

int main() {
//// Step 1 Allocate memory space
int* A, * B, * C; // Host vectors
int* d_A, * d_B, * d_C; // Device vectors
int size = SIZE * sizeof(int);

// Step 2 --> Allocate and initialize host vectors
A = (int*)malloc(size);
B = (int*)malloc(size);
C = (int*)malloc(size);

// Step 3 --> Allocate device vectors
cudaMalloc((void**)&d_A, size);
cudaMalloc((void**)&d_B, size);
cudaMalloc((void**)&d_C, size);

// Step 4 --> initialize the inputs
for (int i = 0; i < SIZE; i++) {
A[i] = i;
B[i] = SIZE - i;
}
cudaMemcpy(d_A, A, size, cudaMemcpyHostToDevice);
cudaMemcpy(d_B, B, size, cudaMemcpyHostToDevice);

// Step 5 --> Launch the Vector Add CUDA Kernel 2 blocks
vectorAdd <<<2, 1024 >>> (d_A, d_B, d_C, SIZE);

// Step 6 --> Copy result back to host
cudaMemcpy(C, d_C, size, cudaMemcpyDeviceToHost);

printf("\nExecution finished\n");

for (int i = 0; i < SIZE; i++) {
printf("%d + %d = %d ", A[i], B[i], C[i]);
printf("\n");
}

// Step 7 Cleanup
cudaFree(d_A);
cudaFree(d_B);
cudaFree(d_C);
free(A);
free(B);
free(C);

return 0;
}
005 levels of parallelization - Vector addition with Extra-large vectors18:31

Query the device properties using the Runtime APIs18:46
Explain how the Cuda runtime apis provide a high level interface to query gpu properties, enumerate devices, and print metrics such as name, memory, and compute capability.
Nvidia-smi and its configurations (Linux User)27:30
The GPU's Occupancy and Latency hiding52:31
Allocated active blocks per SM (important)16:54
how many blocks can we run concurrently per SM?
Starting with the nsight compute (first issue)9:01
All profiling tools from NVidia (Nsight systems - compute - nvprof ...)4:35
Error checking APIs30:16
Nsight Compute performance using command line analysis39:21
Graphical Nsight Compute (windows and linux)59:00

Vector Reduction using global memory only (baseline)1:03:06
Explore the vector reduction baseline using global memory, summing a vector via a tree-based, multi-step process across threads and blocks with partial sums and kernels.
Understanding the code and the profiling of the vector reduction40:02
Explore implementing and profiling a multi-stage vector reduction in cuda, validating gpu results against a cpu reduction, with cpu and device memory management, kernel launches, and profiling insights.
Optimizing the vector reduction (removing the filter)22:45
The Race Condition and the debugging option24:38
Examine the race condition in cuda parallel reductions and how the debugging option (-G/-g) and device optimizations alter output across linux and windows using nvcc.
Optimizing the thread utilizations on vector reduction37:34
Increase thread utilization in a CUDA vector reduction by assigning more elements per thread, reducing grid size, and applying a two-step reduction to boost occupancy and performance.
Optimization using shared memory and unrolling32:20
Shuffle operations optimizations49:09
Explore how shuffle operations enable inter-thread communication within a warp to optimize vector reductions, leveraging registers and shared memory to boost CUDA kernel performance on NVIDIA GPUs.

Requirements

C and C++ basics
Linux and windows basics
Computer Architecture basics

Description

This hands-on course teaches you how to unlock the huge parallel-processing power of modern GPUs with CUDA. You’ll start with the fundamentals of GPU hardware, trace the evolution of flagship architectures (Fermi → Pascal → Volta → Ampere → Hopper), and learn—through code-along labs—how to write, profile, and optimize high-performance kernels.

This is an independent training resource. It is not sponsored by, endorsed by, or otherwise affiliated with NVIDIA Corporation. “CUDA”, “Nsight”, and the architecture codenames are trademarks of NVIDIA and are used here only as factual references.

What you’ll master

GPU vs. CPU fundamentals – why GPUs dominate data-parallel workloads.
Generational design advances – the hardware features that matter most for performance.
CUDA toolkit installation – Windows, Linux, and WSL, plus first-run sanity checks.
Core CUDA concepts – threads, blocks, grids, and the memory hierarchy, built up with labs such as vector addition.
Profiling & tuning with Nsight Compute / nvprof – measure occupancy, hide latency, and break bottlenecks.
2-D indexing for matrices – write efficient kernels for real-world linear-algebra tasks.
Optimization playbook – handle non-power-of-two data, leverage shared memory, maximize bandwidth, and minimize warp divergence.
Robust debugging & error handling – use runtime-API checks to ship production-ready code.

By the end, you’ll be able to design, analyze, and fine-tune CUDA kernels that run efficiently on today’s GPUs—equipping you to tackle demanding scientific, engineering, and AI workloads.

Who this course is for:

For any one interested in GPU and CUDA like engineering students, researchers and any other one

Mastering GPU Parallel Programming with CUDA: ( HW & SW )

What you'll learn

Explore related topics

Course content

Introduction to the Nvidia GPUs hardware12 lectures • 2hr 52min

Installing Cuda and other programs4 lectures • 22min

Introduction to CUDA programming8 lectures • 1hr 52min

Profiling9 lectures • 4hr 18min

Performance analysis for the previous applications2 lectures • 45min

2D Indexing2 lectures • 1hr 16min

Shared Memory + Warp Divergence2 lectures • 50min

Debugging tools1 lecture • 40min

Vector Reduction7 lectures • 4hr 30min

Roofline model1 lecture • 43min

Requirements

Description

Who this course is for: