Available languages:Italiano,

Contents

  1. What is this all about?
  2. What does it look like for users?
  3. Why is apport disabled by default?
  4. How to enable apport
  5. I'm a developer. How do I use these crash reports?
    1. Report format
      1. Fields
    2. Tools
  6. How does it work internally?
    1. Crash interception
      1. Example
    2. Backend
    3. Frontend invocation
  7. Launchpad-based auto-retracer
  8. Per-package Apport Hooks
  9. Use the source, Luke!
  10. Future plans
  11. Further links

What is this all about?

Debugging program crashes without any automated tools has been pretty time consuming and hard for both developers and users. Many program crashes remain unreported or unfixed because:

Apport is a system which:

We are sure that this will lead to a much better level of quality assurance in the future.

If you want to make crash reports of your software even more useful when being reported through apport, please see /DeveloperHowTo.

What does it look like for users?

The user side of apport is designed to be extremely simple and as unannoying as possible.

If any process in the system dies due to a signal that is commonly referred to as a 'crash' (segmentation violation, bus error, floating point exception, etc.), or e. g. a packaged Python application raises an uncaught exception, the apport backend is automatically invoked. It produces an initial crash report in a file in /var/crash/ (the file name is composed from the name of the crashed executable and the user id). If the crashed process belongs to the user who is currently logged in, or it belongs to a system process and the user is an administrator, apport informs the user about the crash and offers to report the problem:

You can click on "Show Details..." to see what data it collected:

If the user leaves the "Send error report" checkbox enabled, Apport uploads the collected information to the bug tracking system. After that it opens the packages' bug filing page with a sensible default bug title and leaves the rest of bug filing process to the web UI.

Why is apport disabled by default?

Apport is not enabled by default in stable releases, even if it is installed. The automatic crash interception component of apport is disabled by default in stable releases for a number of reasons:

  1. Apport collects potentially sensitive data, such as core dumps, stack traces, and log files. They can contain passwords, credit card numbers, serial numbers, and other private material.

    This is mitigated by the fact that it presents you what will be sent to the bug tracker, and that all crash report bugs are private by default, limited to the Ubuntu bug triaging team. We can reasonably expect developers and technically savvy users, who run the development release, to be aware of this and judge whether it is appropriate to file a crash report. But we shouldn't assume that every Ubuntu user of stable releases is able to do so. In 12.04 and up this is transparently handled by whoopsie, see ErrorTracker.

  2. During the development release we already collect thousands of crash reports, much more than we can ever fix. Continuing to collect those for stable releases is not really useful, since
    • The most important crashes have already been discovered in the development release.

    • The less important ones are not suitable for getting fixed in stable releases (see https://wiki.ubuntu.com/StableReleaseUpdates

    • Asking users to send crash reports to us is insincere, since we can't possibly answer and deal with all of them.
  3. Data collection from apport takes a nontrivial amount of CPU and I/O resources, which slow down the computer and don't allow you to restart the crashed program for several seconds.

Note apport does not trap SIGABRT signals. If you are getting such a signal, then please see DebuggingProgramCrash.

How to enable apport

Apport itself is running at all times because it collects crash data for whoopsie (see ErrorTracker). However, the crash interception component is still disabled. To enable it permanently, do: 

sudo nano /etc/apport/crashdb.conf

... and add a hash symbol # in the beginning of the following line: 

        'problem_types': ['Bug', 'Package'],

To disable crash reporting just remove the hash symbol.

I'm a developer. How do I use these crash reports?

Report format

apport internally uses the standard Debian control syntax for reports, i. e. keeps everything in a flat file that looks like this: 

DistroRelease: Ubuntu 12.04
ExecutablePath: /usr/bin/gcalctool
Package: gcalctool 5.8.24-0ubuntu2
ProcCmdline: gcalctool
ProcEnviron:
 SHELL=/bin/bash
 PATH=/usr/sbin:/usr/bin:/sbin:/bin:/usr/bin/X11:/usr/games
 LANG=de_DE.UTF-8
StackTrace:
 [...]
 #0  0x00002ae577bb37bf in poll () from /lib/libc.so.6
 No symbol table info available.
 #1  0x00002ae57786991e in g_main_context_check () from /usr/lib64/libglib-2.0.so.0
 No symbol table info available.
 [...]
CoreDump: base64
 eJzsXQmcFMXV7+XGA0dBREVoDxSPXQYEB...

Only a tiny subset of the available fields are shown here. Apport reports include a core dump in a compressed and encoded format, which is useful for post-mortem debugging and post-mortem generation of a symbolic stack trace.

However, when uploading the data to a bug tracking system, a different format can be used. e. g. when using Launchpad, the data is uploaded in Multipart/MIME format so that the small parts land directly in the bug summary and the big parts become separate bug attachments.

Fields

Some fields warrant further details:

Tools

There are several tools available for working with a crash report:

How does it work internally?

Crash interception

Apport uses /proc/sys/kernel/core_pattern to directly pipe the core dump into apport: 

$ cat /proc/sys/kernel/core_pattern
|/usr/share/apport/apport %p %s %c
$

Note that even if ulimit is set to disabled core files (by specyfing a core file size of zero using ulimit -c 0), apport will still capture the crash.

For intercepting Python crashes it installs a /etc/python*/sitecustomize.py to call apport on unhandled exceptions.

Example

Apport is even able to capture core files if PID 1 (Upstart) dies:

  1. If Upstart detects an internal inconsistency, it raises the SIGABRT signal.

  2. The Upstart crash handler is called on SIGABRT.

  3. Upstart crash handler forks a child process.
  4. The Upstart child process re-raises the signal which results in the child exiting abnormally.

  5. The kernel detects the child process has exited abnormally and calls apport, piping the core file to apports standard input (due to /proc/sys/kernel/core_pattern).

  6. apport writes the core file to disk in /var/crash/.

  7. PID 1 waits for its child to terminate (which only happens once apport has finished writing the core file).

  8. PID 1 exits.
  9. kernel panics.
  10. On next boot, Whoopsie will detect the crash file and process it.

Backend

In order to keep the delay and CPU/IO impact as low as possible, /usr/share/apport/apport only collects data which has to be acquired while the crashed process still exists: information from /proc/pid, the core dump, the executable path, and the signal number. The report is written to /var/crash/executable_path.uid.crash.

Frontend invocation

In Gnome, update-notifier keeps an inotify watch on /var/crash. Whenever there is something new, it calls /usr/share/apport/apport-checkreports. If there are new reports, it calls /usr/share/apport/apport-gtk, which is the frontend shown in the screenshots above.

The frontend then collects additional information like package versions, package file checksums, or OS version, and calls all matching package hooks.

To disable this, you can run gsettings set com.ubuntu.update-notifier show-apport-crashes false (as your ordinary desktop user).

Launchpad-based auto-retracer

The Canonical data center runs a service which automatically retrace bugs with apport. By tagging the bugs according to architecture in Launchpad, a retrace will be done and the tag will be removed. Tags that are used are need-i386-retrace or need-amd64-retrace. See the announcement.

Per-package Apport Hooks

It is possible for packages to specify information gathered from the system and included in the bug report. These are done by apport hooks contained in packages. For some useful examples see:

in /usr/share/apport/package-hooks. There is also a list of packages providing apport hooks.

Please see /DeveloperHowTo for further information.

If a crash or bug report is submitted through apport, the relevant hooks will be run automatically. If you have an already reported bug that was filed without apport, and you are interested in the information from those hooks, you can ask the bug reporter to use apport-collect bugnumber (see #Tools).

Use the source, Luke!

Future plans

Further links

Apport (last edited 2017-05-25 20:03:48 by penalvch)