Available languages: Italiano,

Introduction

This document describes how to install debug packages on Ubuntu releases that do not support debuginfod out of the box, allowing one to provide information regarding providing detailed information when manually performing:

1. A Backtrace.

2. A Valgrind, if the program crashes with a "Segmentation fault" or "Bus error".

3. An Strace.

Using debuginfod

If you are on Ubuntu Jammy (22.04) or later, you do not need to worry about installing debug symbol packages anymore. The Ubuntu project maintains a Debuginfod server, and GDB and other debuginfo-consumer applications support it out of the box. For more information about it, please refer to this page.

Using apport-retrace

If you want to debug a crash in packaged Ubuntu software, Apport will usually pick it up, create a .crash report in /var/crash/ and report the crashed program. From there it is easiest to use apport-retrace.

apport-retrace regenerates the stack traces (both the simple and the threaded one) from an apport crash report from the included core dump, or gives you a Terminal window with gdb in an environment with debugging symbols, reproducing the situation of the crash through the core dump. For this it figures out the set of necessary packages and their accompanying debug symbol packages, so that the regenerated stack trace will be fully symbolic and thus become much more useful for developers to fix the problem.

apport-retrace has two modes: By default it will just regenerate traces based on the packages which are currently installed in the system, i. e. it assumes that all necessary debug symbols for the report are installed (see the next paragraph for this). When specifying the -S option, it creates a temporary "sandbox" and downloads and installs all necessary packages and debug symbols there. It will not do any changes to your system and does not require any special privileges.

Please see the apport-retrace manpage for details and some examples how to run it.

You can also invoke this directly from the Apport crash notification, by clicking the "Examine locally" button:

This will collect some package information and then ask you in which mode you want to run apport-retrace:

Normally you want to keep the first option, which will get you in a gdb session in the sandbox:

If you do not want an interactive gdb session but just want it to update the already existing .crash file with fully symbolic stack traces, you can select the third option.

Installing debug symbols manually

If you want to debug a crash from an application provided by Ubuntu, you are developing yourself, provided by a third-party, or need the debug symbols for particular libraries very often, it is helpful to install the relevant debugging packages.

Built-in debug symbol packages (*-dbg)

For many, but not all, packages, one can simply add a -dbg suffix to the package name to install it. For example:

sudo apt-get install xserver-xorg-core-dbg

Non-built-in debug symbol packages (*-dbgsym)

For additional debug symbols that are not built-in, one must add an additional repository. Please take care to not install both debug symbols for a given package name, as this may cause problems. Your package manager will not check for this overlap and prevent it from happening. For example:

libprotobuf-c1-dbg
libprotobuf-c1-dbgsym

Here are the steps to add this repository:

1. Create an /etc/apt/sources.list.d/ddebs.list by running the following line at a terminal:

   echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list.d/ddebs.list

2. Stable releases (not development, alphas, or betas) require two more lines adding to the same file, which is done by the following terminal command:

   echo -e "deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse\ndeb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list.d/ddebs.list

   You may also add these lines using the Synaptic Package Manager:

3. Choose Synaptic via the System > Administration menu.

4. Choose Software Sources or Repositories via the Settings menu, and click on the Third-Party Software tab.

5. Click the Add button and enter each deb ... line as above one by one and click the Add Source button (you will have to add these lines one at a time).

6. Import the debug symbol archive signing key. On Ubuntu 18.04 and later:

   sudo apt install ubuntu-dbgsym-keyring

   On earlier releases of Ubuntu:

   sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys F2EDC64DC5AEE1F6B9C621F0C8CAB6595FDFF622

   • Note: The GPG key expired on 2021-03-21 and may need updating by either upgrading the ubuntu-dbgsym-keyring package or re-running the apt-key command. Please see Bug #1920640 for workaround details if that does not work.

7. Then run:

   sudo apt-get update

   to update your package list or click the Reload button if you used the Synaptic Package Manager.

8. The debug symbol packages have the '-dbgsym' suffix attached, so to install the debug symbols for the yelp package, you first run:

   apt-cache policy yelp

   This will show you the version number currently installed (we'll use 2.22.1-0ubuntu2.8.04.1 in this example). NOTE: yelp (which is the package name of "Help and Support" located under the 'System' menu) is just an example, and would have to be replaced by the name of the package you want to debug.

9. Install the debug symbols:

   sudo apt-get install yelp-dbgsym=2.22.1-0ubuntu2.8.04.1

   You can also use the Synaptic Package Manager to search for yelp-dbgsym and install it from there.

Installing dbgsym packages from a PPA

If you are debugging a PPA package, have no fear! You can still install -dbgsym packages for them. Where normally you would have a line like:

•  deb http://ppa.launchpad.net/TEAM/PPA/ubuntu RELEASE main

Simply add main/debug at the end, like so:

•  deb http://ppa.launchpad.net/TEAM/PPA/ubuntu RELEASE main main/debug

Then apt update and apt install the -dbgsym packages you want.

Manually finding required debug symbols

The above procedure will install the debug symbol package for yelp only. For many packages, they use libraries shared between different packages. Hence, in order to obtain a readable/more readable stacktrace, or perform other debugging tasks, one must find the debug symbols for these shared libraries. For example, if one had attached gdb to an application and got the following output:

Reading symbols from /usr/lib/x86_64-linux-gnu/libffi.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libffi.so.6

This would indicate that the relevant debug symbols are not installed. However, this by itself wouldn't necessarily be enough information to know which package exactly needs the debug symbols. Hence, one may go to http://packages.ubuntu.com/ and paste into the "Search the contents of packages" keyword box:

x86_64-linux-gnu/libffi.so.6

choose the appropriate Distribution from the drop down, and click Search. This would reveal that the package is libffi6. A check in Synaptic reveals the debug package libffi6-dbg.

Automatically installing all debug symbols

You can use the 'find-dbgsym-packages' command from the 'debian-goodies' package to find debug symbols for a core file, running PID or binary path.

For a binary path it only finds debug symbols for the actual binary itself, and not any dynamically linked library dependencies or other libraries loaded at runtime. For that functionality to work you need to use either a core file or running PID which is the preferred method.

This tool will find both "-dbg" and "-dbgsym" style packages. However it only finds debug symbols for apt repositories that are currently enabled and updated, so you need to ensure that you enable at least the "ddebs.ubuntu.com" archive as described elsewhere on this page and for a Launchpad PPA or the Ubuntu Cloud Archive you need to add another source line with the component changed from "main" to "main/debug".

apt install debian-goodies
find-dbgsym-packages [core_path|running_pid|binary_path]

If the above tool is not working for you, there is an older shell script that attempts to do this by parsing the gdb output instead however often does not work correctly.

You can download the following shell script (list-symbols-packages-v2.1.sh) to resolve all the dependencies. Attaching a debugger to an already running process may require elevated privileges even if you own the process. The following invocation will print out the list of files to install.

sudo bash ./list-symbols-packages-v2.sh -p $(pidof yelp)

To automatically install them apt can be called with the input from the script. The script is invoked with -t for a terse output without the descriptions of the packages and error messages are suppressed:

sudo bash ./list-symbols-packages-v2.sh -t -p $(pidof yelp) 2>/dev/null | xargs -d $'\n' sudo apt-get install 

Version 2 of this script makes it compatible with the newer GDB (which no longer loads all libraries at startup by default). -v2 currently only works if you run it against a currently-executing binary (i.e., with '-p $(pidof <whatever>'). The older version of the script is also available.

Uninstalling all debug symbols

You can remove all debug symbol packages and the ddebs repositories with the following commands:

sudo apt-get remove \.*-dbgsym \.*-dbg
sudo rm /etc/apt/sources.list.d/ddebs.list
sudo apt-get update

The Xorg server

Depending on the circumstances, X server will trap its own crashes and dump a stack trace in /var/log/Xorg.0.log. However, this stack trace is modified by the signal handler itself. If apport is not catching the crash, add this to your /etc/X11/xorg.conf:

Section "ServerFlags"
        Option "NoTrapSignals" "true"
EndSection

Please see X/Debugging for how to debug Xorg server crashes.

Info for the BugSquad

If you're trying to apport-retrace a crash report from a bug that didn't happen on the same Ubuntu release as the one you're running, it would be easiest to just virtualize the environment it is reproducible in. However, if this would not be available to you, one may perform the following example. Say that you're running Trusty and the crash happened on Precise:

1. This will create a minimal Precise system:

   sudo mkdir -p /chroots/precise
   sudo debootstrap precise /chroots/precise/

2. Now change into this minimal precise system:

   sudo chroot /chroots/precise

3. Edit /etc/apt/sources/list and add all the repositories you need, including the ddeb repository.

4. Execute the following in a terminal:

   sudo apt-get update; sudo apt-get install gdb apport

5. use apport-retrace as you're used to.

For more information on dealing with bug reports Apport see Bugs/ApportRetraces.

CategoryBugSquad CategoryDebugging