Linux Diagnostics And Troubleshooting

Learn system monitoring with cockpit, a Red Hat open-source web tool, to monitor cpu, memory, disk io, and logs via a web browser and manage Linux servers graphically.

Discover how performance co-pilot (pcp) collects and queries subsystem data to troubleshoot latency and performance issues, using pmcd, pmstat, vmstat, pm info, and pm logger for historical data.

Learn to set up a centralized rsyslog server with syslog on CentOS 7, forwarding client logs to a central destination for easier troubleshooting and log management.

Configure a centralized log server, enable TCP and UDP logging, and define a cron-focused log template to collect client logs under host name directories while validating ports and firewall rules.

Set up a centralized log server to collect and filter cron and user logs, configure rules and facilities, restart services, and verify log routing from client workstations.

Explore aid, the advanced intrusion detection environment, which monitors file and directory integrity by creating a database snapshot and flagging changes against predefined rules for permissions, Linux context, and attributes.

Learn to verify file and directory integrity with aid by creating a snapshot database, detecting changes or new files, updating the database, and monitoring permission changes.

Learn to audit a Linux system with auditd, create on-the-fly and permanent watch rules to monitor a directory and file changes, and search the audit log for events.

Learn to identify hardware on a Linux system using commands like ls cpu, /proc/cpuinfo, meminfo, dmidecode, lspci, sdparm, and lsusb to diagnose CPU, memory, disk, USB, and PCI hardware issues.

Learn to report hardware errors on Linux using the log tool and machine check exceptions for memory and CPU faults. Install, start, and monitor the service, or use rest daemon.

Install memtest86+, update grub, reboot to access the memory testing boot option, and run exhaustive tests that scan each memory slot for errors to identify faulty memory.

Explore Linux kernel module management by loading and unloading drivers (like E1000), viewing loaded modules with lsmod, and inspecting details with modinfo.

Learn to set kernel module options, load modules with modprobe, and create permanent configurations in /etc/modprobe.d with options for parameters like buffer sizes.

Practice lab session walks through troubleshooting a linux box with a mega raid sas module, showing how to disable msi_x_disable to fix degraded performance and related error logs.

Explore how the Linux storage stack handles IO from application to storage, including VFS, block layer, IO schedulers, and multipath, and learn to recover corrupted file systems and encrypted data.

Understand the virtual file system (vfs) and the common file model that lets standard system calls access ext4, xfs, and nfs across file systems, with page, inode, and dentry caches.

Create and mount file systems such as xfs and ext4 on storage, and explore how device mapper and lvm map blocks across disks with multipath access.

Identify and repair file system corruption by using journaling concepts and tools, recognizing inode and io errors, and choosing appropriate file systems like ext2/ext3/ext4, XFS, or ZFS.

Learn to diagnose and repair ext3 and ext4 file systems with e2fsck, performing dry checks, unmounting when needed, and using backup superblocks to recover metadata.

Explore how to check and repair a zfs file system using zfs_repair, including dry-run mode, unmounted state, automatic use of backup super blocks, and handling corrupted super blocks.

Join a lab session to recover from LVM issues by practicing troubleshooting of volume groups, physical volumes, and logical volumes, using archive and backup files for volume group restore.

restore a volume group and its logical volume from a pre-execution configuration file using cfg restore, then activate the volume and verify the size returns to 100 MB.

Unmount and deactivate the volume group, export it, and move the disk to another system; then import, activate, and mount the vg to verify data.

Explore recovering metadata of a physical volume in an lvm setup, including the header, uid, and volume group configuration entries, using backups from /etc/lvm and related tools.

Recover a corrupted disk header and lvm metadata in a volume group by recreating the physical volume with the same unique id, restoring from backup, and using vg cfg restore.

Configure an iSCSI target and initiator on Linux using target CLI, create back stores and storage objects, and map disks to clients via IQN.

practice lab on iscsi target and initiator demonstrates creating block storage, luns, and acl mappings using the target cli, installing iscsi packages on the client, and preparing initiator connections.

Configure an iSCSI target on the server, discover and login from the client, map the lun, and verify storage with lsblk while managing iSCSI services.

Identify and fix rpm dependency issues and repair the corrupted rpm database using yum and rpm commands, with practical troubleshooting and log checks in /var/log/messages.

learn how to recover a corrupted rpm database by backing up /var/lib/rpm, verifying with rpm db verify, rebuilding Berkeley database and index files, and using rpm db load when needed.

Understand how to use yum history list and yum history info to track package install and removal, undo and redo transactions, and audit changes on a Linux system.

Explore how to use yum history to revert (undo) and repeat (redo) transactions, including installing and removing packages like Vsftpd, with transaction ids and history details.

Identify and resolve network connectivity issues using ping and ICMP, verify access to hosts via DNS or IP, and understand ping options, interface use, and timeouts for troubleshooting.

Learn to use Nmap, the open source network mapper, to scan for live hosts and open ports, audit http and ftp services, and test firewall rules across subnets.

Master nc to communicate with a remote service over tcp or udp, including ssl, and test interactions with a mail server on port 25.

Explore iptraf-ng, a simple ip traffic monitor that tracks inbound and outbound traffic, interfaces, and udp and icmp details. Use its interactive view to monitor tcp connections.

In this practice lab, a server’s second ethernet interface lacks an IP due to an incorrect device name. Correct the interface name in the ifcfg file and reload network manager.

Understand persistent ethernet device naming in Linux, including how udev rules generate names like ENP... or NS33, and how to override or disable them via 70-persistence-net-rules and ifcfg files.

Learn to disable the consistent network device naming in Oracle seven to restore traditional ethernet naming, by editing grub to add net.ifnames=0 (and biosdevname=0 if needed), removing the 80-net-name udev rule, and rebooting to apply changes.

Explore how the network manager daemon uses nmcli to manage ethernet interfaces, check status, configure IPs and gateway settings, add or delete interfaces, and update settings in sysconfig network scripts.

Learn to manage IP version networking on Linux with nmcli commands, check interface status, add IP addresses, and activate or deactivate connections.

Capture tcp/ip packets with tcpdump, the default Linux package, on a chosen interface and save them for later analysis. Use options to filter traffic and analyze captured data.

Learn to use tcpdump to capture packets on a specific interface, limit the count, save to a .pcap file, and read it back with -r, including port 22 filtering.

Understand the Linux boot process from BIOS and MBR through stage one and stage two boot loaders, GRUB, kernel, init, and runlevels defined by init tab, to aid troubleshooting.

Understand the Linux boot process from BIOS to GRUB, kernel, and init, and explore runlevels 0–5, including default 3, graphical 5, and rc.d start or kill scripts.

Explore grub2, the grand unified bootloader, from bios handoff to kernel loading, detailing stage one and stage two bootloaders, root filesystem mounting, and kernel selection via the grub menu.

Grub2 generates grub.cfg from /etc/grub.d and the default file using grub2-mkconfig, with the latest kernel at index zero in the final /boot/grub2/grub.cfg; do not edit grub.cfg directly and keep backups.

Explore grub as an operating system independent boot loader with grub2’s multi-boot support. Understand how grub.cfg is generated with grub2-mkconfig and how grub2 loads the kernel and initramfs.

Learn to boot a Linux system from the grub menu, edit kernels with E for rescue or emergency mode, and use grub commands to boot.

Protect grub by applying a password to prevent access to single user mode and the grub console, using grub-md5-crypt to generate the encrypted password and update grub configuration.

Learn to troubleshoot boot issues when the initramfs is missing or corrupted, using rescue mode, chroot, and methods to recreate the initramfs.

Understand how a missing grub configuration file blocks boot, then boot from grub by selecting the boot partition, kernel, and init ram fs, then restore grub.conf or use rescue mode.

Erasing the MBR on the boot disk prevents booting. Recover by booting from a CD ISO in rescue mode, adjust boot priority, and use dd.

Recover a corrupted MBR by booting into rescue mode, mounting the system, and repairing the MBR with grub-install on the boot disk or via grub setup on the first partition.

boot xl7 into rescue mode, equivalent to single user mode, requiring the root password; use grub2 or edit the linux line to rescue.target, noting local mounts and no networking.

Boot into troubleshooting mode using installation media to reset the root password when rescue or emergency modes cannot boot the system.

reset the root password by booting into troubleshooting mode via grub edits with rd.break, remounting root as read-write, and changing the password, optionally disabling SELinux enforcing to avoid relabeling.

Repair file system issues at boot, including corrupted file systems and non existent devices or UUIDs in fstab, and troubleshoot in the emergency shell with no fail and daemon reload.

Repair boot-time file system issues by fixing fstab mount points, using emergency mode, viewing logs with journalctl -b, remounting the root as read-write, then rebooting and reloading the systemd daemon.

Boot into rescue mode, mount the root file system as read-write, remount, edit fstab, reload the systemd daemon, and reboot to restore normal startup.

Explore SELinux in a Linux diagnostics lab by troubleshooting Apache access, switching between enforcing and permissive modes, and setting proper file context to allow web content delivery.

Explore using the con command to change file and directory labeling, differentiate temporary relabels from permanent policy changes with semanage and restorecon, and observe default labeling behavior.

Explore linux booleans to troubleshoot ftp connectivity, enabling the ftp_home_dir boolean to permit local users to access ftp directories, and review vsftpd logs for denial messages.

Learn to locate Linux logs such as messages and audit.log on the server, and use the sundry Linux troubleshooter in graphical mode to analyze SELinux alerts and access denied events.

Explore pluggable authentication modules (pam) and how they sit between Linux applications and the native authentication system to unify and secure user login across ssh, ftp, and apache.

Understand how PAM centralizes authentication with a stack of modules configured per service under /etc/pam.d, processed top to bottom to determine pass or fail.

Explore pam modules and configurations in linux diagnostics and troubleshooting, including how modules in lib and lib64 perform authentication, lock accounts after failed logins, and verify users against passwd file.

Demonstrate pam.d configuration basics: four entry parts—module interface, control flag, module name, module argument—and the four pam interfaces—auth, account, session, password—and their roles.

Explore PAM control flags that govern authentication by examining module interfaces such as auth, account, session, and password, and learn how required, sufficient, and optional flags shape execution order.

Explore pam modules in Linux, including /etc/pam.d configurations and pam_unix.so for authentication, and place pam_deny at the end with correct 64-bit or 32-bit module paths.

Complete the last lecture of the Linux Diagnostics and Troubleshooting course module by using the Q&A form or pinging the instructor, and good luck with your exam and career.