Parallel computing using MPI

Explore high-performance computing concepts, from clusters of nodes with shared or distributed memory to pipelining and synchronization, and compare massively parallel, embarrassingly parallel, and granularity effects on speed-up.

Explore parallel computer architecture, from stand-alone systems with multiple execution units to networked clusters. Compare tightly coupled shared-memory multiprocessors with loosely coupled distributed-memory clusters and their communication over high-speed networks.

Use portable hardware locality to extract per-node hardware details, visualize cluster topology with lstopo, and combine XML data to optimize parallel processing by reducing communication delays.

Explain the two main parallel computing models—shared memory and message passing—and show how tightly coupled architectures use shared memory, while distributed memory architectures rely on message passing.

Explore parallel algorithm models such as data-parallel, task-parallel, master-slave, pipeline (producer-consumer), and hybrid approaches, and learn how data or tasks are divided among processes to design efficient parallel solutions.

MPI is a standard for message passing in distributed memory systems and a universal parallel programming paradigm, enabling one-to-one and group communication across clusters and heterogeneous networks.

Learn the MPI program structure, including header inclusion, variable scopes, MPI initialization, querying rank and size, performing computation and communication, and finalizing the MPI environment.

Learn to compile and run a simple MPI sample program, where each process prints its rank and the total processes, and how oversubscribing allows more processes than cores.

Explore MPI as SPMD, where a single program runs on multiple processes with distinct ranks, assigning tasks like addition, subtraction, and multiplication while sharing the same core.

MPI terminology explained: identify processes by rank, use communicators to group processes, and exchange data with send and receive calls, buffers, counts, datatypes, and tags.

Explore MPI data types, from basic types like int and char to user-defined derived types, and learn to send both contiguous and non-contiguous data using MPI_Send with a communicator.

Compare mpi programming with client-server programming, showing that mpi uses a single program executed across multiple processes, while client-server requires separate client and server programs, with coordinated parallel execution.

Explore MPI point-to-point communication with blocking and non-blocking send/receive, and bidirectional send-receive; manage status and request objects while using buffer, count, datatype, dest, source, tag, and communicator.

Learn how MPI blocking send and blocking receive enable two processes to exchange an integer data using a shared communicator, illustrating sender and receiver roles.

Demonstrate non blocking send and blocking receive in MPI with sender (rank 0) and receiver (rank 1). The example transmits an integer (50) using a tag and request object.

Demonstrate blocking send and non blocking receive in MPI, with process zero sending and process one waiting for data using a request object and wait to prevent garbage values.

Learn to implement non blocking send and non blocking receive in MPI, using request objects and wait to ensure data arrival, avoid garbage values, and coordinate between sender and receiver.

Learn how to implement a bidirectional send and receive in mpi, exchanging data between processes using tags, status, and a common communicator in a practical example.

discover how MPI collective communication functions enable group communication among all processes in a communicator, using blocking operations like broadcast, scatter, gather, and barrier.

Broadcast data from a root process to all peers with the MPI broadcast function, where the root initializes the buffer and every rank receives the same value.

Explore the MPI scatter function, distributing a data buffer from a root process to multiple receivers, with unique received values per process and an example using four processes.

The MPI gather function collects one element of data from each process to the root, assembling values like 10, 11, 12, and 13 into a single array.

Learn how the MPI reduce function collects data from all processes in a communicator, applies the specified operation at the root process, and stores the reduced result there.

Explore the MPI alltoall function, a collective where every process sends data to all others and receives data from all, demonstrated with a three-process example.

Explore parallel search using MPI by dividing data among processes and scattering elements to each process's subarray. Collect results at the center process and display the number and its index.

Learn to implement a parallel sum with MPI by distributing array elements across processes, each computing a partial sum, then process zero collects and combines them for the final total.

Learn how to count prime numbers in an array with MPI by scattering data across processes, each checking primes in its chunk and using reduce to obtain the total.