Hands on Debugging in C and C++

In this course we are going to focus on examples in GDB [https://www.sourceware.org/gdb/]. The GNU Debugger (i.e. GDB) is a free debugger available on almost every platform and it is incredibly powerful!

Note that the techniques you learn and will be shown with GDB can be used in other debuggers like lldb [https://lldb.llvm.org/] and many of the commands are also the same as GDB (lldb to gdb command map - https://lldb.llvm.org/use/map.html).

Other folks may also want to stick to VSCode, or perhaps Visual Studio (if you're on windows), or even other IDEs. That will be okay but, I think you'll get the most out of this course if you follow along with GDB.

Note: Some other tools that we use may also be Linux specific (e.g. strace or valgrind), but equivalents usually can be found for windows and mac. Feel free to ask for advice in the Q&A.

• Install GDB

  • Ubuntu

    • sudo apt-get install gdb

  • CentOS

    • sudo yum install gdb

• install LLDB (Note: I recommend GDB for this course)

  • Ubuntu

    • sudo apt-get install lldb

A link to my free YouTube lesson on building GDB from source: https://www.youtube.com/watch?v=QFFU1Y8tRV4

In this lesson I am going to show you a few ways to get started on windows. I primarily recommend that you use GDB from within the Windows Subsystem for Linux (WSL) -- it truly provides a great debugging experience on the terminal.

• Instructions on install WSL on windows

  • https://docs.microsoft.com/en-us/windows/wsl/install (Or a google search 'install WSL on windows')

  • Then once Ubuntu is setup, do 'sudo apt-get install gdb'

Some alternatives for debugging would be to follow along in this course through an IDE like Visual Studio. That said, you will need to translate some of the steps a bit, and do some learning on your own.

MinGW is a third alternative, though it does not provide the text user-interface (TUI) mode.

LLDB is one of the most prevalent debuggers on MAC. Many folks might prefer it to GDB for example, though this course primarily uses GDB.

If you'd like, I have the following resources for using LLDB that may be helpful for getting started on this course.

• Free lesson for getting started in 11 minutes: https://www.youtube.com/watch?v=v_C1cvo1biI

• A useful command map of GDB to LLDB commands: https://lldb.llvm.org/use/map.html

The debugging that we are going to talk about in this course is concerned with using a debugger, and actively debugging our program while it is running. This is 'run-time' debugging. Catching errors before the program however, happens at 'compile-time' before our program starts. The distinction is important, and can help you when searching the web for help, and otherwise give you the proper vocabulary to discuss bugs in your program.

Compiler errors happen during the 'parsing' stage of compiling your code. These errors happen at compile-time and must be fixed before we can run our code.

In this lesson I want to show you that while compilers are good at identifying syntactically correct programs, they're not always good at understanding our intent. This can lead to mischievous bugs that are difficult to detect.

Sometimes our compilers can provide us warnings (e.g. using the -Wall flag), and sometimes we can use defensive programming mechanisms ( if(7==x) versus if(x==7)) to write safer software. But then again, it is easy to introduce bugs, so we must learn more techniques!

Compiler warnings are not errors, but they are warnings of code that could cause an error when the program starts running. 

• Tips

  • Compile with -Wall

    • This will display all warnings the compiler knows about.

  • Compile with -Werror

    • This treats warnings as errors, and it is likely the right thing to do to be extra careful.

  • Compile with -Wconversion

    • Treats implicit conversions (casting) as a warning.

• See more warning options here: https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html

It may not always work, but it can be useful to leverage multiple compilers, for example using clang and gcc. They may be able to give you a different set of warnings to help you track down pesky bugs!

Note: Sometimes you may need to use a specific compiler for your project, and it's probably not worth risking doing a build on a wrong tool. So take this as just another trick that may be useful in some sitautions.

One of the first steps to avoiding creating bugs is to write your code neatly. Indentation errors can make your own code harder to read, or someone else. Do make use of your editors refactoring tools, or otherwise get in the habit of writing nice code!

• A tool like 'indent' on Unix can be used to quickly style code.

• Other tools like clang-tidy are also great for refactoring code, and provide better styling options to match your teams or projects coding standards.

In this lesson I've given you some buggy code in c. The goal is to be able to fix this program, and swap the contents of the two arrays. However, there are multiple problems--how are you going to go about fixing the actual program?

printf debugging is a common technique used in many languages (console.log, or std::cout, writeln, etc.).

It's an 'okay' technique that allows you to get useful information out about what your program is doing and the state of a specific object. However, there is a cost of having to recompile your programs, which can be very expensive!

For object-oriented languages like C++ it can often be useful to have for example 'print()' member functions to display the 'state' of your object.

In this lesson I take one more look at debugging using printf, but this time using the C and C++ preprocessor to turn on our logging on and off. You'll often see this in codebases as a way to log information for debug builds as opposed to release builds. This can again be an 'okay' technique, though it is requiring us to make modifications in the source code. As we use our debugger, we'll want to move away from making modifications to our source code.

The Basics of starting and executing GDB.

1. Help page: man gdb

2. Starting gdb: gdb ./prog

   1. Checking your version: gdb --version

   2. Note: You can start gdb with 'gdb --silent' to avoid the information.

   3. Within GDB

      1. print $_gdb_major

      2. print $_gdb_setting_setting(“pagination”)

         1. Gives us the settings that we have

3. Quit GDB

   1. 'quit' or 'q' will terminate GDB

4. Starting gdb with arguments

   1. gdb ./prog my_program_arguments_as_normal

      1. Note that it is important that the program you compiled uses -g to retrieve debugging symbols.

   2. Can use show args and set args to chang the arguments.

5. Start our program

   1. run

6. Start our program but pause at the entry point (i.e. main function)

   1. start

7. list

   1. Shows a listing of where we are in our code (with about 10 lines of code)

8. next or n

   1. Execute the next line of source code

9. step or s

   1. Execute the next instruction, and jump into the function.

10. continue or c 

    1. Run program until a breakpoint is hit, an interrupt (e.g. Ctrl+C), or the program completes execution.

Tip: Pressing 'enter' will repeat the last command that you have typed in.

Note:

• stepi and nexti also allow you to move through assembly instructions, but we will be focusing on C in this course.

When we compile our programs in C, C++, DLang, etc. we will compile with the -g flag. This adds 'symbol' information to our binaries that we execute, as well as information like the line number of file that the symbol originates. Having this information makes it much easier to work with a tool like GDB.

• Another tip is to turn off optimizations with -O0 (capital letter o followed by zero)

• Note: 

  • Can you debug a 'release build' without debug symbols? 

    • The answer is yes, but it will become more difficult--often you will have to loop at the assembly code to do this.

  • Can you debug an 'optimized' debug build?

    • Yes, but again optimizations may change the code around enough such that line numbers do not match up--so be careful, and compile with -O0 (upper case 'oh' and a zero) to be safe.

In this lesson we learn how to 'inspect' values by printing them and the addresses. Now you can finally remove all of the 'printf' and 'cout' statements in your code.

• Lists source code

  • l or list

    • You can type out list linenumber to show a specific line number as well.

  • (set listsize number to change how much source code is displayed)

• Print a value

  • p or print and the value

    • Note: We can even evaluate expressions in our program which is pretty neat (e.g. p i+j where i and j are variables in our program)

    • Note: We can even print out values of pointers using the '*' operator.

    • Note: We can even print out addresses using the '&' operator.

• continue or c

  • This will continue executing your program until a breakpoint is hit.

Pro Tip: Sometimes hitting 'ctrl+l' (i.e. ctrl +  lowercase 'L') will help you clear the screen if the interface gets messy.

Pro Tip: When launching gdb --silent will suppress the additional messages for GDB at the prompt.

When debugging another programmers source code, it can sometimes be helpful to figure out the variables type.

• Return the type of a variable

  • whatis symbol

    • e.g. whatis A (Where 'A' is some variable)

    • e.g. whatis main (Where 'main' is a function)

    • e.g. whatis l->root (Explore a filed of a struct)

  • ptype symbol

    • ptype will do almost the same thing as whatis, but it will unroll any typedefs to give you more information.

  • info types

    • Shows all of the types that are found in your binary

  • info scope location

    • More general command that will find variables within a certain scope, and then you can further query.

      • info scope main

        • Prints all the variables found within main.

We can pause our programs execution just before we execute a line of code using a breakpoint.

• Set a breakpoint

  • br linenumber

  • br functionname

  • br sourcefile:linenumber

• Execute until we hit the next breakpoint

  • 'continue' or 'c'

• View your breakpoints

  • info breakpoints

• Deleting a breakpoint when you do not want it to be available.

  • delete breakpoint_number

Note: There are other types of breakpoints as well.

• temporary breakpoint (tbreak) which will only break once, and then remove itself.

• hbreak a hardware breakpoint, which may be needed if you are using embedded breakpoints. 

The Text User-Interface (TUI) is an excellent way to use GDB from the terminal to read your code as you execute commands in GDB.

• You can launch gdb in TUI mode using gdb --tui ./prog 

  • Alternatively: layout src

    • help layout - will retrieve all the other layouts possible

• Navigation

  • Ctrl-x  1

    • View source and GDB commands

    • (Note you can just type: layout next to rotate between different layouts. Most often we will just use the source and command windows.)

      • focus cmd - focus on command window

      • focus src - focus on the source window

  • Ctrl-x a

    • Toggle between TUI and Command Line mode

  • Ctrl-x o

    • Cycle between windows

• Commands

  • Ctrl+p and Ctrl+n

    • Cycle between commands

  • Up/Down arrows

    • Cycle between commands if in the command-line window

Other navigation notes:

• winheight src -2

  • Shrink the window height

Often times your program may be logging lots of information to standard output which can cluttter the debugging experience. In this lesson, we attempt to fix that by redirecting our output.

• Redirect with -tty

  • Open a second terminal and type 'tty'

  • This tells you where the terminal is connected to for standard input.

• Within gdb type 'tty location_of_terminal_from_tty_command`

• Redirect to another file by simply running with the output redirection

  • (Within GDB)

  • run > outputfilename.txt

• 
  

And that's it! Now you can use GDB more cleanly.

This is exactly like a regular breakpoint, but we have a condition associated with our breakpoint. This can be a useful technique so that we avoid having to hit 'next' or 'step' many times, especially if we have huge loops to go through.

• Here's an example where we break at line 17 if the value x is greater than 100

  • break 17 if x >100

  • Note -- you can also use 'C style syntax' to create more complicated expresssions

    • e.g. break 9 if i < 100 && i > 50

Note: Conditional breakpoints may impact your performance, so if you are executing a graphical application, understand that conditional breaks are constantly being checked and may lower performance. This is something to keep in mind anytime you are using a debugging tool.

We can run our code until a given location more expicity.

• advance

  • We then need to specify a location, either a line, an explicit location (e.g. a function name), or and address

    • advance 10

    • advance foo

    • More help on locations: https://sourceware.org/gdb/onlinedocs/gdb/Specify-Location.html#Specify-Location

• until

  • Is another useful command t hat can run until a location is reached, or until the current stack frame returns.

• skip

  • Occasionally you will jump into some file that you do not care about, or otherwise is a library

  • You can skip these files by specifying which file to skip

    • skip filename

    • info skip will provide details on what has been skipped (skip delete will remove the previously set skips).

• jump or 'j'

  • The jump command allows you to jump to a specific memory in location to start executing.

Instead of setting many breakpoints around our code, we can instead 'watch' a specific variable so that our program will 'pause' every time that variable is modified.

• watch variablename

• rwatch variablename (Stops if the address of the variable is read)

One trick that can be useful is to save your breakpoints

• Save your breakpoints

  • save breakpoints filename

  • source filename - Reload the breakpoints

  • Note: That if you edit your code, some of the breakpoints may move (i.e. the line numbers have changed).

    • IDEs typically manage this for you, and this is one advantage of IDEs which move breakpoints as you add or remove lines of code.

• Breakpoints can also be enabled or disabled

  • enable breakpoint number

  • disable breakpoint number

On occasion you may want to display a variety of information to your screen. Printing repeatedly can get tedious, so the 'display' command is very useful for helping you display variables or information each step of the way.

• display [format] variable_name

  • e.g. display p

  • e.g. display /b p (byte format)

• info display

• undisplay number

  • e.g. undisplay 1 (this removes the first item from 'info display')

Now that you have learned a few things, I want to show you how to get help within GDB. You can always refer to my other videos, but here are a few tricks.

• Type man help on the terminal to get a listing of general commands.

• Type help within GDB to get a listing of the help commands.

• Type 'info' when you start gdb.

  • Try some commands like:

    • info source (once the program starts running)

• The apropos command is useful for 'querying' based off of a keyword to find help if you cannot remember the exact help command.

In this lesson I explain to you some of the basic components of an executable file (Or sometimes an executable object file, or a 'binary'). It's important to understand some of the makeup of a process when debugging, as you may often find yourself having to understand different types of memory and memory errors.

We often think about our programs execution in terms of 'functions', and functions are a key abstraction for us to actually get work done. In this lesson, I describe what is going on in our call stack, and the information that is stored in a 'stack frame'. It's important to understand the call stack, because it becomes a critical tool for understanding how to debug -- it answers the question how did we get here.

• Continue until we return from the current function in the stack frame.

  • 'finish'

• Navigates us up the call stack after we have a crash

  • 'up' or 'u'

• Navigate us down the call stack

  • 'down' or 'd'

• Display the whole stack trace

  • 'backtrace' or 'bt'

• Retrieve current stack frames argument values

  • info args

    • Useful to see what input to function is.

  • info frame

    • Retrieve information about the frame.

  • info locals

    • Information on the local variables.

Segmentation faults are...good. They are designed to keep our operating system safe from other processes. Finding segmentation faults however, is the difficult part. Segmentation faults happen when you access memory that does not belong to your process.

Try to find the bug in this program using gdb!

Let's go through the solution!

Memory leaks occur when we do not free a resource that we have allocated. They can be very subtle with commands like 'strdup' that allocate.

We can use a memory checker like 'valgrind' to find these types of errors.

Note: How bad are memory leaks? You will notice the recording freeze for about a minute as our system becomes unresponsive! (I thought this was quite funny! :) )

In this lesson we are going to learn more about how stack buffer overflows occur. This is in part a review to the program stack, and will hopefully help you build an intuition about memory safety in a process.

• Optional Activities:

  • Stack buffer overflow

    • https://en.wikipedia.org/wiki/Stack_buffer_overflow

  • Review the wikipedia article on the call stack at your convenience:

    • https://en.wikipedia.org/wiki/Call_stack

  • Review stack-based memory allocation

    • https://en.wikipedia.org/wiki/Stack-based_memory_allocation

Try to spot the memory leak in the leak_exercise.c file provided.

Here I'll show you how to run both valgrind and the address sanitizer to fix our memory lek.

It's something that you naturally do--narrow down and find your bug. You can think of it like a binary search for the root cause within your source files (OR your data files as well)

In this lesson we learn about the assert statement and static_assert (available in c++ 11)

• assert

  • 'asserts' something absolutely must be true at run-time.

• static_assert (C++11)

  • 'asserts' something absolutely must be true at compile-time

  • Note: Many programming languages otherwise have their versions of these asserts

One way to investigate how a program is working, by explicitly playing with the functions from within GDB by calling them. You can call functions from within GDB with the call command.

• e.g. call add(2,2);

  • This may be a neat way to test your assumptions about how some code works.

• A quick tip:

  • Type: list add to quickly find that function.

  • Type: info functions to see all of the functions available

It can be very helpful to attach to a running process, especially if a process is long running (i.e. in an infinite loop).

• gdb -p <process id>

• You can run detach when you have completed debugging

If an application has frozen, it can also be helpful to attach to it and see if you can even fix the program.

What is a core dump? A core dump (or core file) is a snapshot of the memory of a program when it crashed (i.e. a fault occurred). You can use core dumps to try to investigate why your program crashed. 

Core files may be useful to share to colleagues to help debug and see if they can spot the issue at the time of the crash (e.g. a bug crash database, or more generally for monitoring the state of a program).  (You can run ulimit -c to see how many core dumps will be created on your unix operating system).

Core Dumps (updating ulimit)

1. ( install: sudo apt install systemd-coredump)

2. Run ulimit -c 

   1. If your value is 0, then change to ulimit -c unlimited

      1. Now your core dumps will be recorded -- though you may want to change this to 0 later on, so as not to fill your machine with core dumps!

3. Compile the sample.c file provided (gcc -g sample.c -o prog)

4. Run ./prog and observe the core file created.

5. Then run coredumpctl gdb

   1. This will load the last core dump file. You can use coredumpctrl dump to further inspect where the core file actually lives if you like.

   2. (Alternatively run gdb -c corefilename)

6. Now you can investigate what went wrong using regular GDB command!

Note: Ctrl+'\' can force a program to terminate and generate a core dump if you want a core of the program.

More information on core dumps:

• https://en.wikipedia.org/wiki/Core_dump

In this lesson, we'll use the gcore tool to create a core file of a running process or generate-core-file filename and also update ulimit There are a few different workflows we'll go through.

Core Dumps Using gcore

1. Start a process and obtain the process id.

2. Then run gcore pid (where pid is the process id from step 1)

3. Then run gdb -c corefilename (or gdb -core corefile)

   1. This will open up the core file.

4. What you probably want though, is the actual exectuable, so run gdb ./prog corefile

   1. Now continue your debugging journey!

Sometimes you have to really take a deep dive into the code and investigate the memory. In this lesson, I'm going to show you how to examine memory.

I'll also show you one more tool as a bonus called hex that is often used to investigate memory.

Resources:

https://sourceware.org/gdb/onlinedocs/gdb/Memory.html

A neat trick you can do is to execute a specific command after a breakpoint has triggered.

1. After you create the breakpoint, type 'commands'

2. Then proceed to type a few commands

3. Then type end when you are finished.

On occasion it may be useful to define your own commands to use in GDB, especially if you have larger tasks that you like to perform.

• Type define command_name

  • Then add as many commands as you like

  • (i.e. try a few commands like print $pc, bt, etc.)

• Then type end 

  • This will terminate the sequence of command that are input.

If you'd like some series of tasks to repeat when you start GDB (or otherwise to define some commands) you can do so in the .gdbinit file (either in a home directory, or the current directory you are executing GDB from)

• Within the current directory where gdb is executing, it can load the .gdbinit file. This way you can put some common 

• You can also use -x to execute specific scripts, so if you do a task frequently you can execute a script file.

From within GDB you can execute many commands from the shell just like you normally would. You can prefix with shell to explicitly do this, and some other commands like 'make' that are common will just execute.

• You can run normal shell commands within gdb if you like

  • shell ls

• You can additionally pipe the output from GDB to shell commands

  • pipe p x | nl

• Some other common commands like 'make' will also work within GDB without having to type 'shell' beforehand (but you can also just do 'shell make')

On occasion you may want to make a change within GDB using your editor of choice. Personally, I prefer to have GDB open in another window, but on occasion you may want to make a quick change.

• First make sure that you set your editor in an environment variable prior to using GDB

  • EDITOR=/usr/bin/vim
    export EDITOR

• Then type 'edit linenumber' or 'edit function' to begin editing the line number or corresponding function within GDB. Changes will be saved, though you may not see the changes within GDB until you reload the compiled file.

• Load a file with 'file' to execute a new file if you need to rebuild (make will typically reload the program)

In this lesson we'll learn about using python within GDB. This is quite a powerful tool again for automation. Many folks will have python already available, so we'll first check to see if it is available. If you do not have Python available, we'll quickly build GDB 13 from source with Python debugging enabled.

In this quick lesson, you will see that GDB actually has a full python interpreter embedded within it. The Python API can provide you another way to pro grammatically use Python to assist with your debugging tasks.

• Examples to try:

  • Try one command with python print("hi") to make sure it works.

  • Let's now try a series of commands

    • python starts the python script

    • print("hello from Python")

    • import gdb (This brings in the gdb api to python)

    • end to terminate the python script

  • Now try some of the API

    • python gdb.execute('start') regularr gdb commands from python

    • python gdb.execute('next')

    • python result = gdb.parse_and_eval('p') // store variable

  • Use Ctrl+C to exit the python interpreter

• Python Interactive interpreter

  • Type pi to enter commands one at a time

• More information on using the API

  • python help gdb

More resources:

1. https://sourceware.org/gdb/onlinedocs/gdb/Python-API.html (Python API in GDB)