Linux Process Monitoring & Diagnostics using /proc interface

In this lecture, you will gain a foundational understanding of how Linux manages processes. Process management is central to Linux's ability to multitask, handling the creation, scheduling, and termination of processes efficiently. You will learn about the different process states (running, sleeping, stopped), and how the kernel uses scheduling algorithms to allocate CPU time fairly across all processes. You will also learn about how various tools to monitor processes on Linux work.

By the end of this lecture, you will gain insight on how various tools used on Linux for gathering process statistics and monitoring work by reading files exposed via /proc filesystem. This knowledge will form the basis for more advanced process diagnostics and performance tuning in Linux systems.

This lecture focuses on the /proc filesystem (procfs), a virtual filesystem that provides critical real-time information about processes and system resources in Linux. The procfs acts as an interface between the kernel and users, allowing users to access detailed process statistics and information, such as memory usage, CPU statistics, and I/O operations, and other resources as files - thus avoiding the need for specific kernel APIs (system-calls) to perform these tasks.

In this lecture, you will understand the concept of pseudo-filesystems in Linux. You will have a high-level understanding on how they are implemented by the Linux kernel and their benefits to the user. You will also learn how the kernel exposes various system and process related information and statistics via the procfs (/proc) filesystem interface.

You will also learn the benefits of procfs and its advantages in building tools and utilities for process and system monitoring in user-space.

In this lecture, we will explore variety of ways to use ps command in Linux to gather process statistics.

You will learn how to use the ps tool to list processes using various selection criteria and also customize the output format being displayed to suit your needs. This can be quite useful if you intend to use ps command within shell scripts for automation work-flows.

By the end of this lecture, you will able to:

• List all processes using ps command

• Understand the significance of the output columns being printed

• List processes based on user, tty, command or select specific processes

• Sort the output of ps command using different sort fields

• Customize the output of ps command suitable for automation within shell scripts

In this lecture, you will learn how to mount procfs filesystem. You will also learn how procfs is mounted by default during Linux boot by the init subsystem.

In this lecture, we will explore some of the key files and directories exposed via /proc by the Linux kernel and understand their significance.

By the end of this lecture, you will be able to:

• Identify the Linux kernel version and release details.

• List of processes currently hosted on Linux system.

• Getting / Tuning kernel's sysctl settings via /proc/sys

• Identify the kernel boot parameters.

• Find out the system uptime and load average.

• Learn the significance of /proc/self

• Control the Linux kernel's dynamic debugging features.

In this lecture, you will learn about the high-level structure and files within the /proc/<pid>/ directory.  You will learn learn about the significance of some of the key files, directories and symlinks within the /proc/<pid>/ directory.

By the end of this lecture, you will be able to:

• Identify the actual program binary executable corresponding to a process

• Identify the current working directory of a process

• Determine if a process is running in a chroot'ed environment

• Identify the environment variables/settings applied to a process

• Identify the command-line arguments provided to the process

In this lecture, you will gain a deeper level understanding on statistics and information of a process as exposed by the Linux kernel via /proc filesystem. You will learn about the significance of some of the key files covered in the earlier lectures.

By the end of this lecture, you will be able to:

• Identify process state, pid, tgid, uid, gid and groups

• Understand the significance of real-uid, effective-uid and saved-uid

• Understand the significance of ppid and tracerpid

• Identify number of threads in a process

• Understand the significance of process umask

• Understand the basics of pid namespaces and how the show up in /proc/<pid>/status entries

In this lecture, you will learn about the various files in /proc/<pid>/ directory that exposes memory usage information and statistics of a Linux process. You will learn about the virtual memory usage, physical memory usage and virtual address mappings  by reading files in the /proc/<pid>/ directory.

By the end of this lecture, you will be able to:

• Identify the peak and current virtual memory usage of a process

• Identify the peak and current physical RAM usage of a process

• Determine if a process has locked memory regions using mlock() API

• Identify the process heap, text, stack and data segments

• Lookup the process segments that include shared library mappings in the process address-space

In this lecture, you will learn how to check and gather CPU usage statistics of a process. You will learn about process CPU utilization in user-mode and kernel-mode and convert the values from jiffies to seconds. You will also learn about CPU affinity of a process, last CPU that executed the process and also determine if a process is more CPU-bound or IO-bound by looking up the context-switch statistics.

By the end of this lecture, you will be able to:

• Determine the overall CPU usage of a running process in kernel-mode and user-mode

• Identify the CPU that executed the process recently

• Identify the CPUs on which the process is set to run on (CPU affinity)

• Determine how CPU-intensive the process is - based on context-switch statistics

In this lecture, you will learn how to gather I/O statistics of a process via /proc filesystem interface. You will learn about read and write statistics on I/O streams (that include files, sockets, pipes and other forms of I/O). You will also learn about disk read/write statistics of a process.

By the end of this lecture, you will be able to:

• Determine the overall I/O overhead of a process

• Determine the disk read/write overhead of a process

In this lecture, you will learn how to get information and statistics of threads running in a process address-space. You will identify thread state and CPU usage of each thread of a process and the kernel stack trace for each thread.

In this lecture, you will learn how to get information about files opened by a process and their file descriptor details (open flags, open mode and file position). We will also explore on files mapped into the process address-space (shared-libraries and other memory-mapped files).

By the end of this lecture, you will be able to:

• Identify files open by a process

• Determine sockets, pipes and other I/O streams opened by a process

• Identify the shared libraries and other files that the process depends on

In this lecture, you will learn more about OOM (Out-Of-Memory) score computed by the kernel for each user-space process. You will also learn how to find the current OOM score of a process and how to adjust them for your needs.

In this lecture, you will learn how to gather and monitor CPU usage statistics of a Linux system by understanding the entries found in /proc/stat file. You will be able to determine if your Linux system is CPU intensive in user-mode, kernel-mode or specifically in servicing interrupts. You will also learn about find out the system boot time, scheduler context-switches, number of processes running and also very high-level hardware interrupt statistics.

In this lecture, you will learn how to monitor hardware interrupt statistics on a Linux system. You will also learn how to setup CPU affinity for hardware interrupts on supported SMP-class systems.

In this lecture, you will learn about overall memory usage statistics of a Linux system by understanding fields in the /proc/meminfo file.

In this lecture, you will learn how to monitor disk I/O statistics of a Linux system. You will learn about the /proc/diskstats and /sys/block/<device>/stat file.

By the end of the lecture, you will be able to:

• determine the rate of disk reads and disk writes

• determine the IO wait times and identify potential disk I/O bottlenecks

In this lecture, you will learn how to monitor network interface statistics on a Linux system. You will learn how to gather the rate at which packets and transmitted / received over network interfaces.

You will also learn how to use iptraf tool to gather detailed interface statistics.

In this conclusion lecture, I will guide you towards further exploration on gathering statistics of various Linux statistics that could not be covered within the scope of this training course.