Debugging Linux Kernel in Deep - Part 1

Explore the irqs-off flag to determine whether interrupts are enabled or disabled, observe how entraps state affects interrupt handling, and watch traces when loading and exiting a kernel module.

The lecture explains preemption, time slices, and the need_resched flag, showing how the interrupt handler decides to preempt the running task and reallocate the cpu.

Identify whether kernel code runs in hardirq, softirq, or interrupt context versus process context, using a tracing module and runtime flags.

Understand how Linux kernel preemption is controlled: disable and enable preemption via the preempt current counter, use macros, and navigate config options like preempt voluntary.

Learn how to start and stop tracing a specific function in the Linux kernel, using a dedicated syntax to execute commands on function entry and exit.

Discover how the trace_options file governs global and current tracer options, enabling or disabling patterns, function names, context info, and output formats for linux kernel tracing.

Learn to change options in the options directory, including global and transit options, and enable or disable contact information and trace.

Learn how to disable interrupt information in the function graph tracer to reduce clutter, while understanding how enabling the function graph affects tracing and potential drops.

Enable function tracing and stack traces to reveal who calls a specific kernel function, producing a per-function stack dump for precise call origins, with filters to scope the trace.

Use trace_marker to allow a userspace process to write into the kernel trace buffer and synchronize tracing between userspace and kernel events.

Use trace_marker from userspace to insert traces around printf, inserting markers before and after calls to identify which functions execute between them and reveal synchronization points.

Learn to profile kernel functions using the function tracer, enable function profiling, and read per-CPU results showing function hits and time spent to identify hot functions.

Explore how disabling interrupts in the linux kernel introduces latency to timer and mouse events, and how to use the function graph tracer to capture and monitor maximum latency.

Use the irqsoff tracer to disable interrupts on the current processor, run a ten-second busy loop, and observe max latency and the function responsible for it.

Enable the tracer after boot to measure startup latencies and identify functions invoked during initialization, using kernel command line options to set function graph, max depth, and threshold.

Identify the deepest kernel stack by enabling stack tracing and analyzing max stack size reports. Use backtrace, trace files, and function graphs to diagnose stack growth and prevent overflows.

Enable dump on oops to capture the console ring buffer in ASCII during a panic, revealing backtrace and panic messages for rapid post-mortem analysis.

Learn how to take a snapshot of the tracing buffer without stopping tracing, copying the main buffer to a snapshot file and switching to the spare buffer to continue.

Explore how tracepoints and trace events enable observing kernel function calls and their parameters, using tracepoint macros, callbacks, and ring buffers to emit human readable data.

Enable and observe kernel events using per-event and subsystem enable files, toggle tracing on and off, view human-readable event data and parameters, and understand how to filter and group events.

Explain how the format file describes each event field, including common fields and event-specific fields, with offsets, sizes, and types, and how it governs the logged output.

Learn how the filter file filters kernel trace events using predicates on the format file fields, with logical operators and constants, and how to clear filters to restore default logging.

Explore advanced filter techniques in Linux kernel debugging, using common and event-specific fields to enable or ignore signal subsystem events and observe signal generated and delivered flows.

Apply subsystem filters to kernel tracing events with a filter file, enabling or disabling events across the subsystem, using common values and noting that missing fields retain previous settings.

Enable, disable, and chain event triggers to control tracing, take snapshots, dump stats, and log function traces for specific subsystems, including function calls like open and read.

Explore how Instagram triggers aggregate trace data into histograms, counting system calls by process with keys and values, sorting by head count or return values, and customizing hash table size.

Explore how to use the set_event interface to enable or disable Linux kernel tracing events by listing available events from the available_events file. Enable events by subsystem or event name.

Explore event pid filtering to trace only kernel events from selected processes, learn how to enable events, view which events are enabled, and observe how removing filters broadens tracing.

Explore a practical example of set_event_pid in the linux kernel, enabling and tracing events, printing hello world, and observing system calls and sleep transitions.

Trace boot-time events with trace_event, enable individual events or all subsystems to analyze kernel initialization calls and their impact on boot time.

Learn to trace mkdir system calls by tracking processes and events, observe arguments like mode and file descriptors, and verify return values and hex-octal representations during directory creation.

explain why you cannot change the event format in this linux kernel debugging context due to read permissions and the lack of available functionality.

Examine usb events from the usb 3.0 controller as devices connect and disconnect, and trace which interrupt-driven functions are invoked in response.

Explore how the Linux kernel schedules processes by tracing context switches and wake ups between previous and next tasks. See how the scheduler logs state changes and runnable transitions.

Explore exception events and page faults in the Linux kernel, distinguishing kernel and user faults, tracing fault entries, and observing major and minor faults.

Explore kernel module events, including reference counts, module_get and module_put, and the load and unload lifecycle with logging and system calls in the Linux kernel debugging.

Explore ext4 events and tracing in Linux kernel debugging; learn how i-nodes, journaling, the superblock, and file operations like create, write, copy, move, and delete are reflected in trace events.

Learn how fs events trace open system calls and reveal which processes open which files, printing file names and enabling targeted proc file tracing for kernel debugging.

explain how linux kernel irq events are traced, showing irq entry and exit, softirq handling, and how to filter and identify which hardware or software triggers interrupts.

Explore Linux kernel network events, focusing on the escape data structure that handles every packet sent or received, net device and intercept handler, and the path through the network stack.

Enable linux kernel events on the BeagleBone, use tracing to study read and write transfers, and observe how the cpu frequency subsystem and regulator drive a blinking led.

Learn to use the trace command to interact with the Linux kernel tracer on BusyBox systems like BeagleBone Black and Raspberry Pi, install via apt, and understand its syntax.

Master trace-cmd list to identify available traces, events, and tracer options, learn how to list traces and events, view options, and apply filters or patterns.

Learn how to use trace-cmd stat to check tracing status, see which events are enabled, and verify tracer configuration, function filters, and related outputs.

Explore how trace-cmd start and trace-cmd stop control tracers, plugins, and events. Learn to enable and disable tracing across subsystems in the linux kernel tracing workflow.

Learn to use trace-cmd show to view trace entries and ongoing trace data by enabling subsystems and using snapshot and full-file options to inspect trace files.

Reset tracing with trace-cmd to reset the function graph and subsystem events, verify which events are enabled, observe how tracing affects performance and file access.

Learn to filter kernel tracing with trace-cmd by selecting specific functions with name-based, wildcard, and global matching, tracing multiple functions, and resetting filters efficiently.

Use trace-cmd to filter by pid and by function, exploring function filtering, event tracing, and selective system-call tracing to observe subsystem activity during execution.

Learn to use trace-cmd record to capture kernel events with plugins and buffers, and use trace-cmd report to generate human-readable summaries from the recorded data.

Learn how to use trace-cmd extract to capture data during start-stop, generating a restore file, and use trace command report to inspect options, plugins, and function events.