Launchpad Entry: karmic-qa-increase-apport-coverage
Created: 2009-06
Contributors: BrianMurray
Packages affected:
Summary
The QA team will drive an effort to extend coverage of apport hooks to all core Ubuntu components.
Rationale
As the number of reported bugs increases over time, we need to improve in a simple way the quality of the information attached to the report. In order to do that the best way would be collecting the relevant data for the specific package affected by the bug. This can be done with apport, but we lack a lot of the hooks needed.
User stories
Selina encounters a bug when using rhythmbox and reports a bug directly in Launchpad. When the bug report is reviewed by triagers it is discoverd that it is missing multiple valuable log files. This then results in the triagers asking for that information, Selina providing it and then the bug becoming triaged. This process could have been avoided if only the package had an apport hook!
Harleen reports a bug using apport, the preferred way, about compiz using apport, however the package has no hook and log files are not gathered. Subsequently, this information is requested by bug triagers and requires Harleen to do some extra work. If the package had an apport hook it could have gathered this information when the bug was being reported.
Implementation
Identify packages that would most benefit, determined by the quantity of bugs that would be improved, from a package hook.
Bug Filing Research
It was suggested to look at the 1000 most recent bug reports filed and determine which packages received the greatest volume of those bug reports. This was most easily and accurately done with the ubuntu-bugs mailing list. The mailing list may be considered more accurate as it contains information about what package the bug was initially filed about which is not tracked in Launchpad. Additionally, the Launchpad API provides the ability to search for bug tasks and not bugs.
Looking at the ubuntu-bugs mbox for June 2009 the top 20% (80% had far too many packages) of bug reports in terms of volume for a 1000 bug sampling follow:
- None: 181
- linux - hook exists: 37
- firefox-3.0 - hook exists: 35
- pidgin: 24
- evolution: 22
- nautilus: 19
- xorg - hook exists: 16
- empathy: 14
- rhythmbox: 14
- alsa-driver - hook exists: 14
- update-manager: 13
- totem: 12
- openoffice.org: 12
- xserver-xorg-video-intel - hook exists: 10
- sun-java6: 9
- gnome-power-manager - hook exists: 8
- grub2: 7
- compiz - hook exists: 7
- thunderbird: 7
- ubiquity - hook exists: 6
- network-manager - hook exists: 5
- byobu: 5
- gnome-terminal: 5
- gnome-utils: 5
- usplash - hook exists: 4
- linux-meta: 4
- kdepim: 4
- firefox: 4
- kvm - hook exists: 4
- fglrx-installer - hook exists: 4
- kompozer: 4
- ekiga: 4
- meta-gnome2: 4
- vm-builder: 4
- kdebase: 4
The same stats are available for May 2009 and April 2009.
None refers to bugs not reported about a specific package. However, in the ideal situation - these would have really been reported about a specific package, so tracking what package they have been assigned to might be useful.
Some packages already have an apport package hook as indicated by 'hook exists'.
Clearly there are some packages that having an apport package hook would improve a fair quantity of bug reports. However, there must be some items worth grabbing (log files, config files, hardware information) for bug reports to benefit from the hook.
Package Hook Writing Classes
It is very easy to write a new package hook and to test that it works properly, however it seems that most people are not aware that they can be written or how to write them. Ubuntu QA team members who have written a package hook should document how to write a hook and give a class in #ubuntu-classroom to educate more people, including Bug Squad members, how to write them. The class should also cover how to get them incorporated into Ubuntu.
Unresolved issues
There is not a really good way to discover what package bugs without a package were later assigned. This might be useful information to refine the target packages for apport hooks.
BoF agenda and discussion
Agenda
Inventory and review packages that currently have apport hooks.
- Firefox 3.0
- apparmor
- apport
- compiz
- cups
- fglrx-installer
- printing packages
- gnome-power-manager
- jockey
- kvm
- linux
- xorg packages
- mdadm
- network manager
- nvidia-graphics-driver-*
- pulseaudio
- tracker
- usplash
Packages that might benefit from apport hooks.
- rss-glx
- xscreensaver
- update-manager
- ubiquity
- hal
Reportbug looks in /usr/share/bug/*/presubj to present information to include in bug reports. Can any of these be implemented in apport hooks? There was an apport bug regarding this but it was decided that the reportbug guidelines need manual review. If not are they candidates for package bug filing instructions? Possibly it would be great to have a list of all the packages that ship something in /usr/share/bug.
Other particular areas?
- - packages with debugging procedures?
Usability issue: what triggers apport is a signal that sometimes does not get called. It'd be great to get the correct information when an application freezes and not just crashes.
Statistics on adoption of bugs reported with apport, but not on coverage. Statistics about packages with a lot bug reports that do not have apport hooks.
- Possibility to trigger a data collection event in apport when an application becomes non-responsive.
https://wiki.ubuntu.com/Apport
- contains listing of package hooks and when they came out
- Easy to write new hooks
- What are the blockers to writing new hooks?
- Awareness - people aren't aware of them, or how to create them
- Do upstreams provide enough debugging information for apport to actually collect something?
We need ways to send the information to related packages.
How much in-depth can you achieve with an apport hook?
Pick a target:
- Top bugs filed and by feature freeze they have to have a hook
- Should this be a goal or a policy?
- Look at the top 80% of new bug reports
- We might already be there ITO of package coverage, but we need to increase the number of bugs filed with apport/ubuntu-bug
* Packages with already available debugging information A list is MANUALLY maintained at https://wiki.ubuntu.com/Apport#Per-package%20Apport%20Hooks
- Can we generate a list of packages with apport hooks from the archive?
What should be considered private information in a bug reported by apport? Can a package hook make all bug reports that utilize that hook private?
Target
- Look at latest 1000 bugs reported
- Sliced by package
- Take the top 50%
- Create a new IRC channel? or just use #u-qa? to educate people about writing package hooks
- we already have a couple of channels #ubuntu-bugs - #ubuntu-quality and also #ubuntu-classroom for sessions like this.
How to deal with bugs that apport reports against the wrong package? (i.e. python)
- File a bug for apport about them.
- Cleanup python and other intereperter package bugs with apport-crash tags
- Look at new apport-crash bugs as these have not been triaged and might need reassignment
ACTION ITEMS
- DONE - Schedule a plenary (henrik to speak to pitti about slides) and a workshop
- DONE - Brian to look at the latest 1000 bugs reported
- Tom and Brian to talk about ways to track package reassignment in Launchpad