
Learn remote debugging of the Linux kernel via a client-server setup, using a gdb server on the target and a development machine running Jilib to set breakpoints and inspect memory.
Configure two machines, a host development machine and a target VM running Linux with symbols and debug info, to run kgdb via gdb; ensure kernel patch for mainline support.
Set up two virtual machines (development host and target) with a serial port connection, start the target first, and verify bidirectional serial communication.
Set up two connected virtual machines, compile the kernel with cagily support, configure the target for debug mode, and connect from the development host to the target.
Copy kernel and debug symbols from the target to development machine after downloading the source on both machines. Ensure network connectivity and set the target to run in debug mode.
Enable gdb support in the boot process, attach to the target via gdb, set breakpoints, and verify the boot sequence and driver startup using the chosen bootloader.
Connect to the target or serial port using the target command, then continue execution on the remote target to perform a backtrace and debug the kernel.
Disable kernel address space layout randomization to reveal function addresses during debugging. Use the no casula flag on the target machine, observe backtrace and breakpoint behavior.
Add and verify breakpoints in kernel debugging, step through read operations, observe buffer and file descriptor values, inspect breakpoint hits, and continue to completion.
Switch back into debugging mode by sending the sissako sequence or invoking the exported break point function, enabling breakpoint execution and live kernel debugging.
learn how to enable kgdb at runtime by configuring the serial port baud rate and sending the magic sequence to switch the target into debugging mode, enabling remote kernel debugging.
Explore tui mode in the linux kernel debugger, using a two-window dui interface to display source, registers, and commands with live highlighting of the executing line.
Set and observe breakpoints for creating and deleting directories using a debugger in dui mode, stepping through folder creation operations and inspecting relevant values.
Learn to set breakpoints on a specific line with the syntax break <target>, line number, then run and continue to verify execution at that line; delete breakpoints as needed.
Modify kernel code and build a new kernel image, copy it to the target, and use gdb to set breakpoints, verify exact execution lines, and observe target behavior.
Shows that a single serial port cannot serve both the kernel console and kgdb; demonstrates configuring two serial ports, updating the command line via Grub, and debugging with kgdb.
Agent proxy enables using a single serial port for both console and debugging by muxing into two boards, then connect via localhost ports and observe logs.
Activate kgdb control messages via the command line, configure the console and grub, and connect to a target remote development device to observe kernel debugging messages.
Learn to use kgdbreboot to reboot the target for kernel debugging, control notifications with default value zero or minus one, and set a breakpoint and backtrace to inspect the kernel.
Discover how the out-of-memory killer reviews running processes, selects a memory-hungry victim, signals it to free memory, and how stress provokes the oom killer to observe the kill process.
Enable pretty print in gdb to render kernel structures, improving readability of complex data such as struct file and related outputs.
During a kernel panic, the system enters debugger mode for memory inspection, backtraces, and register checks to diagnose the fault, with four ways to trigger this mode.
Learn to print and inspect kernel variables during debugging, using breakpoints and global symbols to verify values on the target machine, while noting that local variables cannot be printed.
Set and change kernel module variables, like panic on oops and panic on one, to control panic behavior; load or unload the module and rebuild the kernel to observe effects.
Enable remote debugging on the target, verify bidirectional packet exchange, set breakpoints in GDB, and confirm the target remains in debugging mode to diagnose connection problems.
Set a breakpoint on proc cmdline and connect to the target in debugging mode. Print the value of a global variable and step through execution to observe the breakpoint.
Copy the target build, load module symbols with the text segment address, and use breakpoints in the in-tree module to trace packet transmission during kernel module debugging.
Set a breakpoint at the interrupt handler and observe the function execution on the target. Single-step through the interrupt handler and inspect AC to learn debugging interrupts with breakpoints.
learn to use linux kernel gdb scripts to streamline debugging, verify script enablement, load scripts from the target path, and run commands to inspect processes, modules, devices, and filesystems.
Use lx-symbols to load symbols for all modules on the target kernel. It scans modules and loads their symbol files into gdb so you can set breakpoints.
Explore the lx_current macro, which returns the current task, and learn how to query the task on a specific cpu, including its name and state, especially in device drivers.
Debug out-of-tree kernel modules by building the control module, transferring it to the development machine, loading it on the target, and using gdb with symbols for breakpoints.
Debug the init and exit sections by setting breakpoints in kernel/module.c, loading the module on the target, and stepping through module load and delete module cleanup.
Build an image for the Raspberry Pi 3 with the required kernel configuration, update the kernel config, enable the serial port, and deploy the image.
Build and install the sdk by generating a toolchain that includes gdb and the dcc compiler, then install the sdk tailored for the rba image.
Flash a downloaded image onto an SD card for the Raspberry Pi 3 via a VM, then connect the serial port to complete the setup.
Connect the serial port to the Raspberry Pi, set the baud rate to 1150 with minicom or screen, and verify the flashed image and module presence.
Set up an agent proxy to share a single serial port among the console and devices, using localhost and designated ports on the Raspberry Pi 3.
Connect to a Raspberry Pi 3 over GDB for kernel debugging by configuring the VM Linux image with debug symbols, setting the serial port, and attaching via target remote localhost:551.
Configure gdb for Linux kernel debugging by setting substitute-path to the kernel source and pointing it to the Raspberry Pi 3 source folder, then use list to inspect lines.
User space processes can be debugged using gdb. With gdb you can
single-step,
stop,
resume,
put break-points,
inspect memory and variables,
look through call stack information
What if we can use gdb on running Linux Kernel.
Problem:
How can a gdb debug running kernel as it is just a user space process.
Solution:
Client/Server Architecture
User space programs can be debugged remotely using the combination of gdbserver on the target machine and gdb on the host machine/development machine.
The Linux kernel has a GDB Server implementation called KGDB. It communicates with a GDB client over network or serial port connection
In this course we will learn:
How to setup KGDB
Various ways of getting the target into development machine
Setting Breakpoints
Printing and Setting Variables
Using a single serial port for both kgdb and console messages
Getting the kernel messages in gdb window
Debugging Linux Kernel Modules (In-Tree, out of tree)
Use of GDB Scripts present in the Linux Kernel