Mastering GPU Parallel Programming with CUDA: ( HW & SW )

Name: Mastering GPU Parallel Programming with CUDA: ( HW & SW )
Rating: 4.3 (645 reviews)

Performance Optimization and Analysis for High-Performance Computing

Bestseller

Created byHamdy egy

Last updated 10/2025

English

What you'll learn

Comprehensive Understanding of GPU vs CPU Architecture
learn the history of graphical processing unit (GPU) until the most recent products
Understand the internal structure of GPU
Understand the different types of memories and how they affect the performance
Understand the most recent technologies in the GPU internal components
Understand the basics of the CUDA programming on GPU
Start programming GPU using both CUDA on Both windows and linux
understand the most efficient ways for parallelization
Profiling and Performance Tuning
Leveraging Shared Memory

Course content

12 sections • 58 lectures • 23h 3m total length

GPU vs CPU (very important)20:49
NVidia's history (How Nvidia started dominating the GPU sector)5:18
Architectures and Generations relationship [Hopper, Ampere, GeForce and Tesla]16:07
How to know the Architecture and Generation6:56
The difference between the GPU and the GPU Chip4:50
The architectures and the corresponding chips5:25
Nvidia GPU architectures From Fermi to hopper12:24
Parameters required to compare between different Architectures24:05
Please don't skip this video. It is pivotal for the the whole course.
Half, single and double precision operations6:26
Compute capability and utilizations of the GPUs8:30
Before reading any whitepapers !! look at this8:42
Volta+Ampere+Pascal+SIMD (Don't skip)52:08

The course github repo
Mapping SW from CUDA to HW + introducing CUDA.12:49
001 Hello World program (threads - Blocks)20:39
Compiling Cuda on Linux9:54
002 Hello World program ( Warp_IDs)9:03
003 : Vector addition + the Steps for any CUDA project22:30
004 : Vector addition + blocks and thread indexing + GPU performance18:21
#include <stdio.h>
#include <cuda_runtime.h>
#include <device_launch_parameters.h>

#define SIZE 2048 // Define the size of the vectors

// CUDA Kernel for vector addition
__global__ void vectorAdd(int* A, int* B, int* C, int n) {
int i = threadIdx.x + blockIdx.x * blockDim.x ;
C[i] = A[i] + B[i];
}

int main() {
//// Step 1 Allocate memory space
int* A, * B, * C; // Host vectors
int* d_A, * d_B, * d_C; // Device vectors
int size = SIZE * sizeof(int);

// Step 2 --> Allocate and initialize host vectors
A = (int*)malloc(size);
B = (int*)malloc(size);
C = (int*)malloc(size);

// Step 3 --> Allocate device vectors
cudaMalloc((void**)&d_A, size);
cudaMalloc((void**)&d_B, size);
cudaMalloc((void**)&d_C, size);

// Step 4 --> initialize the inputs
for (int i = 0; i < SIZE; i++) {
A[i] = i;
B[i] = SIZE - i;
}
cudaMemcpy(d_A, A, size, cudaMemcpyHostToDevice);
cudaMemcpy(d_B, B, size, cudaMemcpyHostToDevice);

// Step 5 --> Launch the Vector Add CUDA Kernel 2 blocks
vectorAdd <<<2, 1024 >>> (d_A, d_B, d_C, SIZE);

// Step 6 --> Copy result back to host
cudaMemcpy(C, d_C, size, cudaMemcpyDeviceToHost);

printf("\nExecution finished\n");

for (int i = 0; i < SIZE; i++) {
printf("%d + %d = %d ", A[i], B[i], C[i]);
printf("\n");
}

// Step 7 Cleanup
cudaFree(d_A);
cudaFree(d_B);
cudaFree(d_C);
free(A);
free(B);
free(C);

return 0;
}
005 levels of parallelization - Vector addition with Extra-large vectors18:31

Query the device properties using the Runtime APIs18:46
Nvidia-smi and its configurations (Linux User)27:30
The GPU's Occupancy and Latency hiding52:31
Allocated active blocks per SM (important)16:54
how many blocks can we run concurrently per SM?
Starting with the nsight compute (first issue)9:01
All profiling tools from NVidia (Nsight systems - compute - nvprof ...)4:35
Error checking APIs30:16
Nsight Compute performance using command line analysis39:21
Graphical Nsight Compute (windows and linux)59:00

Requirements

C and C++ basics
Linux and windows basics
Computer Architecture basics

Description

This hands-on course teaches you how to unlock the huge parallel-processing power of modern GPUs with CUDA. You’ll start with the fundamentals of GPU hardware, trace the evolution of flagship architectures (Fermi → Pascal → Volta → Ampere → Hopper), and learn—through code-along labs—how to write, profile, and optimize high-performance kernels.

This is an independent training resource. It is not sponsored by, endorsed by, or otherwise affiliated with NVIDIA Corporation. “CUDA”, “Nsight”, and the architecture codenames are trademarks of NVIDIA and are used here only as factual references.

What you’ll master

GPU vs. CPU fundamentals – why GPUs dominate data-parallel workloads.
Generational design advances – the hardware features that matter most for performance.
CUDA toolkit installation – Windows, Linux, and WSL, plus first-run sanity checks.
Core CUDA concepts – threads, blocks, grids, and the memory hierarchy, built up with labs such as vector addition.
Profiling & tuning with Nsight Compute / nvprof – measure occupancy, hide latency, and break bottlenecks.
2-D indexing for matrices – write efficient kernels for real-world linear-algebra tasks.
Optimization playbook – handle non-power-of-two data, leverage shared memory, maximize bandwidth, and minimize warp divergence.
Robust debugging & error handling – use runtime-API checks to ship production-ready code.

By the end, you’ll be able to design, analyze, and fine-tune CUDA kernels that run efficiently on today’s GPUs—equipping you to tackle demanding scientific, engineering, and AI workloads.

Who this course is for:

For any one interested in GPU and CUDA like engineering students, researchers and any other one

Mastering GPU Parallel Programming with CUDA: ( HW & SW )

What you'll learn

Explore related topics

Course content

Introduction to the Nvidia GPUs hardware12 lectures • 2hr 52min

Installing Cuda and other programs4 lectures • 22min

Introduction to CUDA programming8 lectures • 1hr 52min

Profiling9 lectures • 4hr 18min

Performance analysis for the previous applications2 lectures • 45min

2D Indexing2 lectures • 1hr 16min

Shared Memory + Warp Divergence2 lectures • 50min

Debugging tools1 lecture • 40min

Vector Reduction7 lectures • 4hr 30min

Roofline model1 lecture • 43min

Requirements

Description

Who this course is for: