DebuggingProgramCrash

Differences between revisions 1 and 120 (spanning 119 versions)
Revision 1 as of 2005-05-28 20:36:44
Size: 1040
Editor: adsl-213-190-44-43
Comment: imported from the old wiki
Revision 120 as of 2014-12-14 10:37:42
Size: 8653
Editor: penalvch
Comment: RM'ed erroneous / as running command verbatim caused it not to work.
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
= DebuggingProgramCrash = Available languages: [[https://wiki.ubuntu.com/DebuggingProgramCrash_it| Italiano]],
<<Include(Debugging/Header)>>
||<tablestyle="float:right; font-size: 0.9em; width:40%; background:#F1F1ED; margin: 0 0 1em 1em;" style="padding:0.5em;"><<TableOfContents>>||
Line 3: Line 5:
 1. Check if the package has a debugging version available. In general, debugging packages will be named with a `-dbg` suffix
 1. If not, you can generally* build one this way:
  a. Install the build-time dependencies for the package: {{{
sudo apt-get build-dep <package>
This document describes how to debug Ubuntu package crashes and install debug packages on Ubuntu, which will aid in providing information for bugs.

== Using apport-retrace ==

If you want to debug a crash in packaged Ubuntu software, [[https://wiki.ubuntu.com/Apport|Apport]] will usually pick it up, create a `.crash` report in `/var/crash/` and report the crashed program. From there it is easiest to use [[http://manpages.ubuntu.com/apport-retrace|apport-retrace]].

`apport-retrace` regenerates the stack traces (both the simple and the threaded one) from an apport crash report from the included core dump, or gives you a Terminal window with gdb in an environment with debugging symbols, reproducing the situation of the crash through the core dump. For this it figures out the set of necessary packages and their accompanying debug symbol packages, so that the regenerated stack trace will be fully symbolic and thus become much more useful for developers to fix the problem.

`apport-retrace` has two modes: By default it will just regenerate traces based on the packages which are currently installed in the system, i. e. it assumes that all necessary debug symbols for the report are installed (see the next paragraph for this). When specifying the `-S` option, it creates a temporary "sandbox" and downloads and installs all necessary packages and debug symbols there. It will not do any changes to your system and does not require any special privileges.

Please see the [[http://manpages.ubuntu.com/apport-retrace|apport-retrace manpage]] for details and some examples how to run it.

You can also invoke this directly from the Apport crash notification, by clicking the "Examine locally" button:

 {{attachment:apport-examine-locally.png}}

This will collect some package information and then ask you in which mode you want to run `apport-retrace`:

 {{attachment:apport-retrace-gui.png}}

Normally you want to keep the first option, which will get you in a gdb session
in the sandbox:

 {{attachment:apport-retrace-gdb.png}}

If you do not want an interactive gdb session but just want it to update the already existing `.crash` file with fully symbolic stack traces, you can select the third option.

== Debug Symbol Packages ==
If you want to debug a crash in a project you are developing yourself or from a third-party package, or need the debug symbols for particular libraries very often, it is helpful to install those permanently into your system.

First, check if there is a package with a -dbg suffix in the main Ubuntu repositories. These are the debug symbol packages, and are equivalent to '-dbgsym' described below. You can safely use either one, but not both at once. Try installing the package, adding -dbg to its name, for example:
{{{
sudo apt-get install xserver-xorg-core-dbg
Line 8: Line 39:
  a. Build .debs for debugging: {{{
DEB_BUILD_OPTIONS="nostrip noopt" sudo apt-get source -b <package>
}}}
  a. Install the needed .debs (they will be in the current working directory if the build succeeded): {{{
sudo dpkg --install <package.deb>
}}}
 1. Start the program under control of `gdb`: {{{
gdb <program>
(gdb) run <arguments, if any>
}}}
 1. The program will start. Perform any actions necessary to reproduce the crash
 1. Retrieve a backtrace of the crash: {{{
(gdb) thread apply all bt full }}}
if the package in question is the ''xserver-xorg-core'' package.
Line 22: Line 41:
 1. Include the complete output from gdb in your bug report If there is no -dbg package:
Line 24: Line 43:
* most of the packages support the build of debugging version in this way. If that doesn't work for a package you can open a bug against it.  1. Create an `/etc/apt/sources.list.d/ddebs.list` by running the following line at a terminal: {{{
echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list.d/ddebs.list }}}
 1. Stable releases (not alphas and betas) require two more lines adding to the same file, which is done by the following terminal command: {{{
echo "deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list.d/ddebs.list }}}

 You may also add these lines using the Synaptic Package Manager:
  * Choose ''Synaptic'' via the ''System > Administration'' menu.
  * Choose ''Software Sources'' or ''Repositories'' via the ''Settings'' menu, and click on the ''Third-Party Software'' tab.
  * Click the ''Add'' button and enter each ''deb ...'' line as above one by one and click the ''Add Source'' button (you will have to add these lines one at a time).

 1. Import the debug symbol archive signing key: {{{
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 428D7C01 }}}

 1. Then run: {{{
sudo apt-get update }}}
 to update your package list or click the ''Reload'' button if you used the Synaptic Package Manager.

 1. The debug symbol packages have the '-dbgsym' suffix attached, so to install the debug symbols for the package 'yelp', you first run: {{{
apt-cache policy yelp }}}
 This will show you the version number currently installed (we'll use 2.22.1-0ubuntu2.8.04.1 in this example).
 '''NOTE:''' 'yelp' [which is the package name of "Help and Support" located under the 'System' menu] is ''just an example'' and '''not''' a part of the command: 'yelp' is only used to demonstrate the procedure. You must replace 'yelp' with the name of the package you want to debug.

 1. Install the debug symbols: {{{
sudo apt-get install yelp-dbgsym=2.22.1-0ubuntu2.8.04.1 }}}
You can also use the Synaptic Package Manager to search for 'yelp-dbgsym' and install it from there.

The above procedure will install the debug symbol package for yelp only. Chances are that yelp uses shared libraries in other packages and debug symbols for them might be required in order to obtain a readable stack trace or perform other debugging tasks.

You can download the following [[attachment:list-symbols-packages-v2.1.sh |shell script (list-symbols-packages-v2.1.sh)]] to resolve all the dependencies. Attaching a debugger to an already running process may require elevated privileges even if you own the process. The following invocation will print out the list of files to install. {{{
sudo bash ./list-symbols-packages-v2.sh -p $(pidof yelp)}}}

To automatically install them apt can be called with the input from the script. The script is invoked with {{{-t}}} for a terse output without the descriptions of the packages and error messages are suppressed {{{
sudo bash ./list-symbols-packages-v2.sh -t -p $(pidof yelp) 2>/dev/null | xargs -d $'\n' sudo apt-get install }}}

/!\ Version 2 of this script makes it compatible with the newer GDB (which no longer loads all libraries at startup by default). -v2 currently only works if you run it against a currently-executing binary (i.e., with '-p $(pidof <whatever>'). The [[attachment:list-symbols-packages.sh|older version]] of the script is also available.

 a. Now you make a [[Backtrace]].
 a. You can also run [[Valgrind]], if the program crashes with a "Segmentation fault" or "Bus error".
 a. Optionally, you may be asked to produce an [[Strace]].
 a. You can also provide this file : '''~/.xsession-errors'''

=== How do we remove all this stuff after getting the trace and get back to a normal system? ===

You can remove all debug symbol packages and the ddebs repositories with the following commands: {{{
sudo apt-get remove \.*-dbgsym
sudo rm /etc/apt/sources.list.d/ddebs.list
sudo apt-get update }}}

=== References ===

 * Announcement: https://lists.ubuntu.com/archives/ubuntu-devel-announce/2006-September/000195.html

== The Xorg server ==

The X server will by default trap its own crashes and dump a stack trace in /var/log/Xorg.0.log. However, this stack trace is modified by the signal handler itself. To get a "normal" crash, which will trigger a core dump (and ''apport'' reporting), add this to your /etc/X11/xorg.conf: {{{
Section "ServerFlags"
        Option "NoTrapSignals" "true"
EndSection }}}
Please see [[X/Debugging]] for how to debug Xorg server crashes.

== Info for the BugSquad ==

If you're trying to `apport-retrace` a crash report from a bug that didn't happen on the same Ubuntu release as the one you're running, do the following:

Say that you're running `Gutsy` and the crash happened on `Feisty`:

 0. This will create a minimal `feisty` system. {{{
sudo mkdir -p /chroots/feisty
sudo debootstrap feisty /chroots/feisty/}}}
 0. Now you change into this minimal `feisty` system. {{{
sudo chroot /chroots/feisty}}}
 0. edit `/etc/apt/sources/list` and all the repositories you need, especially Martin's ddeb repository.
 0. {{{
apt-get update; apt-get install gdb apport}}}
 0. use `apport-retrace` as you're used to.

For more information on dealing with bug reports [[Apport]] see [[Bugs/ApportRetraces]].

----
CategoryBugSquad CategoryDebugging

Available languages: Italiano,

Debugging Central

This page is part of the debugging series — pages with debugging details for a variety of Ubuntu packages.

This document describes how to debug Ubuntu package crashes and install debug packages on Ubuntu, which will aid in providing information for bugs.

Using apport-retrace

If you want to debug a crash in packaged Ubuntu software, Apport will usually pick it up, create a .crash report in /var/crash/ and report the crashed program. From there it is easiest to use apport-retrace.

apport-retrace regenerates the stack traces (both the simple and the threaded one) from an apport crash report from the included core dump, or gives you a Terminal window with gdb in an environment with debugging symbols, reproducing the situation of the crash through the core dump. For this it figures out the set of necessary packages and their accompanying debug symbol packages, so that the regenerated stack trace will be fully symbolic and thus become much more useful for developers to fix the problem.

apport-retrace has two modes: By default it will just regenerate traces based on the packages which are currently installed in the system, i. e. it assumes that all necessary debug symbols for the report are installed (see the next paragraph for this). When specifying the -S option, it creates a temporary "sandbox" and downloads and installs all necessary packages and debug symbols there. It will not do any changes to your system and does not require any special privileges.

Please see the apport-retrace manpage for details and some examples how to run it.

You can also invoke this directly from the Apport crash notification, by clicking the "Examine locally" button:

  • apport-examine-locally.png

This will collect some package information and then ask you in which mode you want to run apport-retrace:

  • apport-retrace-gui.png

Normally you want to keep the first option, which will get you in a gdb session in the sandbox:

  • apport-retrace-gdb.png

If you do not want an interactive gdb session but just want it to update the already existing .crash file with fully symbolic stack traces, you can select the third option.

Debug Symbol Packages

If you want to debug a crash in a project you are developing yourself or from a third-party package, or need the debug symbols for particular libraries very often, it is helpful to install those permanently into your system.

First, check if there is a package with a -dbg suffix in the main Ubuntu repositories. These are the debug symbol packages, and are equivalent to '-dbgsym' described below. You can safely use either one, but not both at once. Try installing the package, adding -dbg to its name, for example:

sudo apt-get install xserver-xorg-core-dbg

if the package in question is the xserver-xorg-core package.

If there is no -dbg package:

  1. Create an /etc/apt/sources.list.d/ddebs.list by running the following line at a terminal:

    echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list.d/ddebs.list 
  2. Stable releases (not alphas and betas) require two more lines adding to the same file, which is done by the following terminal command:

    echo "deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse
    deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list.d/ddebs.list 
    You may also add these lines using the Synaptic Package Manager:
    • Choose Synaptic via the System > Administration menu.

    • Choose Software Sources or Repositories via the Settings menu, and click on the Third-Party Software tab.

    • Click the Add button and enter each deb ... line as above one by one and click the Add Source button (you will have to add these lines one at a time).

  3. Import the debug symbol archive signing key:

    sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 428D7C01 
  4. Then run:

    sudo apt-get update 

    to update your package list or click the Reload button if you used the Synaptic Package Manager.

  5. The debug symbol packages have the '-dbgsym' suffix attached, so to install the debug symbols for the package 'yelp', you first run:

    apt-cache policy yelp 
    This will show you the version number currently installed (we'll use 2.22.1-0ubuntu2.8.04.1 in this example).

    NOTE: 'yelp' [which is the package name of "Help and Support" located under the 'System' menu] is just an example and not a part of the command: 'yelp' is only used to demonstrate the procedure. You must replace 'yelp' with the name of the package you want to debug.

  6. Install the debug symbols:

    sudo apt-get install yelp-dbgsym=2.22.1-0ubuntu2.8.04.1 

You can also use the Synaptic Package Manager to search for 'yelp-dbgsym' and install it from there.

The above procedure will install the debug symbol package for yelp only. Chances are that yelp uses shared libraries in other packages and debug symbols for them might be required in order to obtain a readable stack trace or perform other debugging tasks.

You can download the following shell script (list-symbols-packages-v2.1.sh) to resolve all the dependencies. Attaching a debugger to an already running process may require elevated privileges even if you own the process. The following invocation will print out the list of files to install.

sudo bash ./list-symbols-packages-v2.sh -p $(pidof yelp)

To automatically install them apt can be called with the input from the script. The script is invoked with -t for a terse output without the descriptions of the packages and error messages are suppressed

sudo bash ./list-symbols-packages-v2.sh -t -p $(pidof yelp) 2>/dev/null | xargs -d $'\n' sudo apt-get install 

Warning /!\ Version 2 of this script makes it compatible with the newer GDB (which no longer loads all libraries at startup by default). -v2 currently only works if you run it against a currently-executing binary (i.e., with '-p $(pidof <whatever>'). The older version of the script is also available.

  1. Now you make a Backtrace.

  2. You can also run Valgrind, if the program crashes with a "Segmentation fault" or "Bus error".

  3. Optionally, you may be asked to produce an Strace.

  4. You can also provide this file : ~/.xsession-errors

How do we remove all this stuff after getting the trace and get back to a normal system?

You can remove all debug symbol packages and the ddebs repositories with the following commands:

sudo apt-get remove \.*-dbgsym
sudo rm /etc/apt/sources.list.d/ddebs.list
sudo apt-get update 

References

The Xorg server

The X server will by default trap its own crashes and dump a stack trace in /var/log/Xorg.0.log. However, this stack trace is modified by the signal handler itself. To get a "normal" crash, which will trigger a core dump (and apport reporting), add this to your /etc/X11/xorg.conf:

Section "ServerFlags"
        Option "NoTrapSignals" "true"
EndSection 

Please see X/Debugging for how to debug Xorg server crashes.

Info for the BugSquad

If you're trying to apport-retrace a crash report from a bug that didn't happen on the same Ubuntu release as the one you're running, do the following:

Say that you're running Gutsy and the crash happened on Feisty:

  1. This will create a minimal feisty system.

    sudo mkdir -p /chroots/feisty
    sudo debootstrap feisty /chroots/feisty/
  2. Now you change into this minimal feisty system.

    sudo chroot /chroots/feisty
  3. edit /etc/apt/sources/list and all the repositories you need, especially Martin's ddeb repository.

  4. apt-get update; apt-get install gdb apport
  5. use apport-retrace as you're used to.

For more information on dealing with bug reports Apport see Bugs/ApportRetraces.


CategoryBugSquad CategoryDebugging

DebuggingProgramCrash (last edited 2022-12-20 22:13:06 by sergiodj)