InfiniBand Deep Dive: Networking for AI focused Datacentres

InfiniBand Deep Dive: Networking for AI Data Centres

Welcome! I'm here to help you truly understand InfiniBand — the high-performance fabric powering the world's most demanding AI and HPC environments.

As AI workloads explode in scale, the network is no longer an afterthought — it is the bottleneck. Slow fabrics mean idle GPUs, longer training times, and wasted investment in expensive compute. This course gives you the deep, practical knowledge to understand, deploy, and troubleshoot the technology at the heart of modern AI data centres.

What you'll learn:

• Why traditional Ethernet and TCP/IP fall short for AI workloads — and how InfiniBand solves latency, throughput, and CPU bottleneck challenges

• The full InfiniBand architecture — Physical, Link, Network, Transport, and Upper layers — with real-world analogies that make concepts stick

• RDMA, Zero-Copy transfers, Queue Pairs, Memory Registration, and GPUDirect RDMA — the core technologies behind high-speed GPU communication

• How the Subnet Manager works — LID assignment, topology discovery, routing table programming, and failover with Standby SM

• Traffic isolation using Partition Keys (PKey) — configuring Full and Limited membership across multi-tenant AI clusters

• Quality of Service (QoS) — assigning Service Levels (SL), mapping to Virtual Lanes (VL), and configuring bandwidth weights in OpenSM

• Routing algorithms in depth — MINHOP, UPDN, Fat-Tree, Adaptive Routing — and why Adaptive Routing is critical for elephant flows in AI workloads

• Congestion control, Credit-Based Flow Control, credit loops, and how to prevent fabric deadlocks

• Fabric monitoring and management at scale using NVIDIA Unified Fabric Manager (UFM), including Cyber-AI and RBAC

• Hands-on troubleshooting using ibdiagnet, ibtracert, mlxlink, ibstat, smpquery, and more

This course is packed with visual analogies, architecture diagrams, and practical troubleshooting scenarios that make even the most complex concepts click. Whether you're a network engineer, a cloud infrastructure specialist, or an AI platform team member, this course will give you the edge to design and operate high-performance AI fabrics with confidence.

No InfiniBand experience required — just bring your curiosity and your ambition.

Let's get started!

NOTE - For learners preparing for NCP-AIN Exam

This course covers 40–50% of the NCP-AIN exam domains, with a focused deep dive on InfiniBand. It is a valuable study companion to cover a significant portion of topics for the certification.