Summary

It is still apparent that the incoming volume of kernel bugs remains problematic to manage. The ratio of incoming bugs to resources still doesn't scale. The goal of this spec is to re-evaluate our current bug management work flow and practices and determine a more effective way to manage kernel bugs.

Release Note

Due to the increasing volume of incoming kernel bugs an improved and sustainable approach to bug management is being introduced. See KernelTeam/KernelLucidBugHandling and KernelTeamBugPolicies for more information.

Rationale

During the Karmic cycle we started implementing Kernel Bug Days as well as writing kernel arsenal scripts to combat the growing volume of kernel bugs. If we stop to look at the numbers, up until Karmic Beta we were seeing much better improvements compared to Jaunty. In the 5 month time span between Karmic opening and Karmic Beta, there was an increase of ~1000 open bugs of which the bugs in a New state increased by ~300. Compare that to the 5 month time span between Jaunty opening and Jaunty Beta, there was a far less total increase of ~300 open bugs but the bugs in a new state increase by ~750.

However, then Karmic Beta landed and we were completely smothered with the increased bug volume. Again, just looking at the statistics, between Karmic Beta and Karmic Final (a time frame of less than 1 month), we saw an increase of ~1500 open bugs of which the bugs in a New state increased by ~1250. Compare that to the time frame between Jaunty Beta and Jaunty Final which saw an increase of ~900 open bugs of which the bugs in a New state increased by ~700.

Then we compound this with the fact that things like kerneloops wasn't disabled until after Karmic's Final release and in the 1 week following Karmic's release, we saw an additional increase of ~1200 open/new bugs.

It's obvious that we need to continue to examine our current bug management work flow and determine how we can effectively deal with this ever growing volume of kernel bugs.

User stories

Design

Some ideas to improve handling new bugs coming in:

  1. Interactive apport hooks for the kernel.
  2. Apport bug reporting improvements.
  3. LP lib scripts to search for known issues with fixes (for ex kerneloops)
  4. Improve arsenal scripts
  5. Leverage the HWDB to find bugs on the same hardware/platforms.
  6. Modify the http://wiki.ubuntu.com/KernelTeam/BugDay

  7. Continue to work on building and mentoring the Ubuntu kernel community.
  8. Additional Crack of the Day kernel builds

Implementation

  1. Interactive apport hooks for the kernel. Note: patch already accepted to apport for Lucid. See bug 444672
    • [DONE] Is the issue confirmed with the usptream kernel?
    • [DONE] Is this a regression?
    • [DONE] If a regression, please note most recent working kernel
    • [DONE] Is this reproducible?
    • [DONE] if reproducible, please describe steps to reproduce
    • [DONE] If not reproducible, how often does the issue appear?
    • [ACTION] Should we add something like "Is there a known patch to fix this issue?" so we know the ones we should be able to close quickly?
  2. Apport bug reporting improvements.
    • [ACTION] Don't allow reporting against linux-meta, just use linux
    • [ACTION] Is it possible to tag bugs based on subsystem (networking, audio, sound, wifi) ?
  3. LP lib scripts to search for known issues with fixes (for ex kerneloops)
  4. Improve arsenal scripts
    • [ACTION] Need to make these smarter so they can be run without intervention for at least bugs reported against lucid
    • [ACTION] go through and tag (w/ no e-mail?) bug reports using release information so tag bugs as karmic, hardy etc...
    • [ACTION] detect and tag bugs against staging drivers (also maybe inject [Staging] in the title of the bug?)
    • See lp:arsenal-devel for current set of scripts and lp:~leannogasawara/arsenal/kernel for additional merge request
  5. Leverage the HWDB to find bugs on the same hardware/platforms.
  6. Modify the http://wiki.ubuntu.com/KernelTeam/BugDay

    • [ACTION] Don't assign lists to specific developers. Have one common list for anyone to pick from.
    • [ACTION] look for interesting bugs with videos/pictures, simply cause they are more interesting
    • [ACTION] Re-evaluate our advertising of bug days
  7. Continue to work on building and mentoring the Ubuntu kernel community.
    • [ACTION] Update/Create wiki, ex "How Can we Help"
      • How to Test
      • How to Triage
      • How to run pre-proposed
      • How to bisect regression
      • Section listing area of kernel (network, audio etc) with pointers to the right people
  8. Additional Crack of the Day kernel builds
    • [ACTION] Lucid daily build of tip
    • [ACTION] build some bisect points between releases
    • [ACTION] add HEAD to /proc/version_signature for tracking purposes

BoF agenda and discussion


CategorySpec

KernelTeam/Specs/KernelLucidBugHandling (last edited 2009-11-30 22:04:31 by c-76-105-148-120)