Ubuntu Wiki

• Launchpad Entry: qa-checkbox-bug-filing

• Created: 2008-12-15

• Contributors: schwuk

• Packages affected: checkbox

Summary

When failures or errors are encountered while performing tests in Checkbox, users should be encouraged to file bugs. Ideally this should be immediately to a) ensure the bug is filed and b) attached relevant information to the bug.

How we handle this from a user experience perspective is the focus of this blueprint.

Rationale

Currently there is no facility for filing bugs within Checkbox. Bugs could discovered in any of the following:

• Checkbox itself
• The test currently being performed
• The subject of the test (e.g. network card or application)

If a failure is encountered for either of the latter two, then the user should be encouraged to file a bug immediately against the relevant package or project.

While this could result in a proliferation of hardware related bugs being filed, it is felt that these would not on the whole be new bugs, just providing visibility to existing ones.

Use Cases

• Paul's network card is not detected by Checkbox, so he is prompted to file a bug against Checkbox for this
• When Sarah runs the sound card test, it fails to work so she is prompted to file a bug against (ALSA/Pulse Audio)

Assumptions

• Bugs will be created in a similar manner to Apport - supporting information is uploaded and the user is taken to a instance-specific bug filing page with some information pre-populated
• That filing bugs in Launchpad will "break" the Checkbox user experience, so this needs to minimised and the users encouraged to continue their testing

Design

When a test fails or a failure is detected, the user will be prompted to file a bug or link to an existing bug.

New Bugs

If they choose to file a new bug, then apport is triggered to collect and upload the relevant information (using the debugging field in the test definition), and then the whole bug filing process is continued in line with current apport processes.

When the user returns to Checkbox, they are prompted to enter the bug number for the bug they have just filed which is then recorded along with the test results.

Existing Bugs

If they chose to link an existing bug, then they are prompted to enter the bug number for the bug they have just filed which is then recorded along with the test results.

No Bug

If they chose not to file a bug, then Checkbox continues.

Should this bug number be validated?

Implementation

• Bugs that are filed via Checkbox will be tagged accordingly
• Regressions (if we can spot them) are also tagged accordingly
• Apport is used to gather debugging information information and trigger the bug filing process in Launchpad
  • The default web browser will be user

UI Changes

• Each test will have an (unobtrusive) "File Bug" button/link
• When a test is marked as failed, the user is prompted to either:
  • File a bug
    • If they agree, they are taken to Launchpad
    • If they decline, the testing process continues
  • Link an existing bug report to this (failed) test
• Checkbox will not proceed until the bug filing process is completed

For more detail, please see the CheckboxUI spec.

Test/Demo Plan

• Users are prompted to file bugs on failures
• Users are able to file bugs at will
• Bug numbers are correctly attached to test results.

Unresolved issues

Can Apport be used in this way - if not we can use the same processes in Launchpad to attach information to the bug.

How will this work on proxied systems?

BoF agenda and discussion

When performing tests, users should be encouraged to file bugs immediately to a) ensure the bug is filed and b) attach relevant information to the bug. How we handle this from a user experience perspective is the focus of this blueprint.

Questions:

* How does this relate to apport? Do we use apport to file the bug or simply use the same LP calls?

• user should have the ability to indicate that they have already filled a bug report
• will use apport and its data gathering (per-package hooks)
• report each bug individually when it happens in the community version
• identify that the bug report was filled when using checkbox via a tag
• when there is a failure 3 options
  • 1) I want to file a bug 2) I know there is already a bug about this 3) I don't want to file a bug

* How do we note that a bug is a regression?

• tag the bug when it is being filled as regression-
  • ideally the tag will be dynamic depending on the release and package version being tested
• store a hash of the test, to identify whether it has changed, when recording results (pass / fail)
• prompt to report bug should have a timeout in case of long running tests such as the X tests

Can apport queue up bug reports for filing later or report bugs from the console, ie in a server environment?

CategorySpec