
Learn perf fundamentals and metrics, using perf stat to measure hardware and software events and cpu utilization; explore perf record, perf report, flame graphs, tracepoints, and perf probe.
Set up a perf development environment on ubuntu 22.04 with kernel 6.5, compile perf from source, install dependencies, and configure perf_exec_path to enable perf scripting and off cpu analysis.
Master perf stat and performance analysis by learning key metrics such as context switch rate, page fault rate, and cpu utilization, across system-wide and process metrics and hardware counters.
Explore how software performance is defined through measurement and metrics like response time, throughput, and resource utilization, and learn to set objective targets and analyze bottlenecks with perf.
Explore matrix multiplication performance by analyzing cache misses in a row-major layout; swap inner loops to improve memory access for A, B, and C, using perf.
Master perf stat by monitoring targets such as the entire system, specific processes, threads, or CPUs, using options A, P, T, C, or a command, with real world examples.
Explore hardware and software performance events with perf, including PMU, tracepoints, and user space probes. Use perf list to categorize, filter, and collect performance data for meaningful metrics.
Use perf stat to count hardware and software events in counting mode for CPU performance. Compare naive matrix multiplication with Strassen's to reveal FP instruction counts versus runtime.
Explore cpu utilization with perf stat to compare single-threaded and multi-threaded matrix multiplication, analyzing time elapsed, user and system time, task clock, cpu utilization, and ipc to optimize performance.
Explore measuring maximum instructions per cycle and CPU frequency with a NOP loop, perf stat, and taskset on specific CPUs. See how frequency affects performance and throughput.
Explore context switches and CPU migration in Linux, and understand how their costs impact performance in multitasking systems.
Explore page faults, distinguishing minor and major faults, and see how transparent huge pages reduce memory overhead and improve performance in Linux with perf.
Explore perf hardware events on Intel CPUs, using modifiers to count kernel and user activity and PMU syntax with raw and PMU formats for branch instructions and conditional branches.
Explore how perf derives symbolic PMU event names by inspecting the Linux 6.5 source code, PMU events, arch/x86 maps, and the mapfile.csv linking CPU models to names.
Measure software energy usage on Linux with perf energy events for cores, RAM, GPU, and CPU package. Compare CPU frequencies using a matrix multiplication workload to balance power and performance.
Explore perf stat basics, metrics, and events to monitor system wide or per process performance, identify bottlenecks, and access hardware counters using raw syntax or the PMU syntax.
Explore micro-architectural performance analysis with perf stat and the top down method to diagnose CPU bottlenecks, covering vector instructions, speculative execution, locks, false sharing, and pgo.
Explore the CPU pipeline and micro-operations, using the top down microarchitecture analysis method to identify bottlenecks via pipeline slots, front end bound, back end bound, and bad speculation in perf.
Explore the TMA metric hierarchy and its six layers, including retiring, bad speculation, back end bound, and front end bound, to identify CPU bottlenecks and guide optimizations.
Explore how perf stat integrates with TMA metrics and metric groups to measure performance. Identify top down slots and TMA retiring within metrics and groups using perf.
Examine retiring category performance, comparing standard qsort with an AVX-512 vectorized quicksort, and showing how reducing light and heavy operations via vector instructions boosts speed.
Analyze branch mispredictions and bad speculation, showing how excessive branching stalls the cpu pipeline, and demonstrate branchless programming with a table lookup to dramatically improve performance measured by perf.
Explore backend bound performance by examining memory bound and core bound stalls, L1–L3 caches, and the impact of atomic locks on latency.
Explore how false sharing on cache lines degrades backend performance through cross-processor cache invalidations and how 64-byte padding can separate fields, reducing contention.
Explore front end bound issues in fetch latency and bandwidth, instruction cache misses, and TLB misses, and how profile guided optimization with bolt boosts Clang performance using perf.
Master micro-architectural performance analysis by applying the top down method to identify retiring, branch misprediction, front end bound bottlenecks, and optimize with vector instructions and branchless programming.
Learn to master performance analysis with perf, covering software performance profiling, perf record and perf stat, flame graphs, workload analysis, and off cpu analysis to identify bottlenecks.
Learn to profile software performance with sampling, instrumentation, and tracing to identify bottlenecks, characterize workloads, and optimize hotspots using perf record and dynamic instrumentation.
Learn performance profiling with perf record and perf report, including event selection and sampling rate. Identify hot paths and bottlenecks with call graphs and data analysis.
Explore real-time cpu profiling with perf top, a blend of perf record and perf report that reveals which functions consume cpu cycles and supports call graphs via -g.
Explore flame graphs to visualize call chains and sample frequencies from perf data, using perf record, perf script, and flamegraph.pl to identify performance hotspots and optimize code.
Analyze memcached workload characteristics by benchmarking and generating flame graphs to reveal how cpu time is spent on sending data, reading requests, and epoll wait, with per-thread insights.
Learn to create custom flame graphs with perf by selecting specific events like last level cache misses, capturing call chains, and pinpointing cache inefficiencies in matrix multiplication.
Explore Linux trace points and how to use perf to diagnose performance issues, including schedulers, context switches, and shed switch trace points, with practical data analysis.
Analyze performance from an off-CPU perspective using perf, visualizing scheduling gaps with flame graphs and tracepoints. Identify IO waits, context switches, and wake-up events to optimize disk IO and locking.
Profile software performance with perf tools, capturing data with perf record and perf report. Monitor real-time performance with perf top and visualize bottlenecks using flame graphs and tracepoints.
Instrument user and kernel code with perf probe to capture real-time performance data and monitor kernel activities, observe memcached at runtime, and craft perf scripts to analyze function latencies.
Learn dynamic instrumentation with perf probe to insert a k probe in kernel or user space, without altering source code, and capture function arguments and return values at runtime.
Explore how to catch crash points in Linux with perf, using the force segfault kernel helper, probes, and register-based data capture to diagnose segmentation faults.
Master debugging memcached with perf probe and perf record by inspecting function arguments, local variables, and struct members, using line and filter options to target specific events.
Learn perf scripting to analyze data with predefined scripts or Python and Perl templates, customize fields, and run real time or scripted performance analysis for apps like memcached.
Learn to measure function latency with perf by creating a Python template script, placing start and end probes, recording timestamps, and analyzing latency with delta time and a histogram.
Explore Amdahl's law to identify performance bottlenecks, focusing on the sequential part and mutex locking's impact on throughput in multi-core systems.
Master dynamic instrumentation with perf probe and perf script on Linux to insert probes into running code without recompilation and analyze targeted performance data.
Measure with perf to baseline Linux performance, visualize bottlenecks with flame graphs, then optimize by addressing sequential code, off-CPU waits, and CPU-bound issues using top-down microarchitecture analysis and Amdahl's law.
Welcome to "Mastering Performance Analysis with perf"—the ultimate course to unlock the full power of Linux's official performance tool, perf. This course is designed to guide you to expert-level performance analysis, equipping you to become a true performance guru in Linux systems.
Developed over two years with extensive revisions, this course delivers essential skills and techniques, ensuring you learn the most effective ways to leverage perf for deep system insights.
Throughout the course, we’ll cover a range of essential tools, methodologies, and technologies, including:
Hardware Event Monitoring: Track events like cache misses and branch mispredictions for deeper insights into hardware behavior.
System Event Monitoring: Monitor critical system events such as context switches and page faults.
Top-Down Microarchitecture Analysis (TMA): Master this technique to write efficient code and maximize CPU performance.
Flame Graphs: Learn how to create and interpret flame graphs to pinpoint performance bottlenecks.
Linux Tracing Frameworks: Explore tracing tools like tracepoints, kprobes, and uprobes to capture in-depth performance data.
Kernel Function Tracking: Gain insights into kernel activities and identify points of failure like segmentation faults.
Application Function Tracing: Track application behavior to diagnose and resolve performance issues.
Custom Performance Analysis: Learn to design custom analyses, such as calculating function latency.
Hands-On Examples: Practice with real-world examples to build practical performance engineering skills.
By the end of this course, you’ll have the confidence to identify, diagnose, and optimize performance issues in your applications and systems as part of your daily work.