Summary

It has become apparent that the incoming volume of kernel bugs has become problematic to manage. The ratio of incoming bugs to resources available will not scale as the volume continues to increase. The goal of this spec is to introduce better bug management workflows and practices to combat this growing number of bugs.

Release Note

Due to the high volume of incoming kernel bugs an improved approach to bug management is being introduced. See KernelKarmicBugHandling and KernelTeamBugPoliciesfor more information.

Rationale

At the time of the Jaunty 9.04 release there were over 5000 open kernel bugs. That's a 1300+ net increase in bugs compared to the number of open bugs when Intrepid released. The kernel team can not realistically be expected to close 5000 bugs in a single release cycle. If the total number of incoming kernel bugs continues to grow at a faster rate than the existing number of bugs can be closed, it's obvious that a new approach to dealing with the incoming bug volume needs to be addressed. Otherwise the probability of a critical or high priority bug not being seen by the kernel team becomes a greater concern. Additionally Ubuntu users will continue to become discouraged when a bug they reported is not addressed. This will likely result in users not reporting bugs at all.

User stories

Research

It seems Fedora has also ran into this same situation and have handled the scenarios similarly to what this spec will outline. I was unable to find any documentation regarding Suse's bug policy.

Implementation

The biggest issue is making sure that the incoming bugs are in an appropriate debug state for the developers to begin working on a bug. It may take weeks of communication back and forth between and reporter and triager before a bug has all the appropriate logs attached. It's been decided that much of this process could be streamlined by using a series of Arsenal scripts. Below is a description of how the Kernel Arsenal scripts will work. The end goal is for a bug reported against the Ubuntu kernel in Launchpad to have the appropriate debugging info attached as well as tested the latest upstream mainline kernel to verify if the issue exists or is fixed upstream. The Ubuntu Kernel Team's focus will shift to those bugs which are confirmed to exist upstream or are fixed upstream but exist in the Ubuntu kernel. We will update the KernelTeamBugPolicies wiki to match the changes being introduced here. It will more thoroughly document the new policies and procedures.

Kernel Arsenal Scripts

arsenal/contrib/linux/process-new-bugs.py

if Status == New
    if has_tag("omit"|"workflow"|"review-request")
        exit

    if has_tag("apport-bug")
        tag "needs-upstream-testing"
        logs_complete = True
        message = test upstream

    if has_tag("apport-package")
        logs_complete = True

    if Importance == Wishlist
        exit

    if bug_tasks > 3
        tag "review-request"
        exit

    if symptom == workflow
        tag "workflow"
        exit

    if symptom == sound bug
        tag "kernel-sound"

    if attachments == 0
        tag "needs-kernel-logs"
        tag "needs-upstream-testing"
        message = apport-collect
    elif parse_attachments()
        logs_complete = True

    if old bug (ie last comment > 120 days ago)
        tag "needs-verification"

    tag "kj-triage"

    if logs_complete
        status = Confirmed
    else
        status = Incomplete

    if message
        post message as a comment to the bug

arsenal/contrib/linux/process-incomplete-bugs.py

if status="Incomplete (with response)" and has_tag("needs-kernel-logs")
    if "apport-collect data" in message_colection:
        remove tag "needs-kernel-logs"
        status = Confirmed
    else
        tag "review-request"
elif status="Incomplete" and no comment for > 120 days
    status = Won't Fix

arsenal/contrib/linux/process-incomplete-bugs.py

if status="Confirmed" and has_tag("kj-triage") and not has_tag("needs-kernel-logs"&&"needs-upstream-testing")
    status = Triaged

Test/Demo Plan

The Kernel Arsenal scripts which should land soon at https://code.edge.launchpad.net/~arsenal-devel. They have a dryrun option enabled to ensure bugs are handled appropriately before turned on.

Additionally we will target small subsets of the overall number of kernel bugs until it's deemed reasonable to run the scripts for every kernel bug. This will then be handled by a cron to regularly run the Kernel Arsenal scripts.

Unresolved issues

BoF agenda and discussion

Below are a few ideas that have been suggested (some more drastic than others):

A new Kernel's bug bankrupcy policy was proposed for treating kernel bugs.

  1. Bugs reported against Linux package at LP must be tested against latest upstream kernel. If they're still valid, a new upstream bug report must be openand at LP an upstream bug watch must be put in place.
  2. If no response is received after 30 days, the Bug would be marked as Won't Fix.
  3. This policy would be discussed with Jono in order to get the Community input.
  4. The upstream bug report must include the kernel version.

a few ideas discussed at the meeting:

CategorySpec

specs/KernelKarmicBugHandling (last edited 2009-06-12 18:34:10 by c-76-105-148-120)