Launchpad Entry: karmic-qa-checkbox-enhancements
Created: UDS Karmic
Contributors: DaveMurphy
Packages affected: checkbox, checkbox-cli, checkbox-gtk
Summary
This spec tracks the high priority enhancements to Checkbox for Karmic. They are designed to improve the user experience and provide required functionality for other Checkbox related blueprints.
Low priority/undefined enhancements are dealt with in ../CheckboxWishlist
Rationale
Checkbox requires various enhancements to improve the user experience, or provide functionality for teams using Checkbox. These enhancements vary in size and complexity, but none are significant enough to require their own blueprint. Therefore a decision was made to combine them into a single blueprint for tracking purposes.
Enhancements
Each enhancement is dealt with individually.
Apport Hooks
Checkbox currently does not have any hooks for Apport to use when reporting bugs. Following issues during the Jaunty development cycle which resulted in a number of duplicated bugs, these should be added early in the Karmic cycle to address any future issues.
Hooks should be provided for:
- version
- submission.xml
- logs (if available, see "Logging By Default" below)
In addition there should a general apport hook to include the system/submission ids if they are cached (see below).
Use Cases
- Sam is performing system testing during the Karmic development cycle, and the sound test triggers a crash. Apport will be notified of the crash, and will prompt Sam to file a bug with a appropriate information attached.
- Mike encounters an crash when trying to play a video using Totem. When he files the bug using Apport, the bug is linked with his most recent hardware submission for this machine.
Implementation Details
Logging By Default
For users running the development release and using the extended checkbox-qa package we should enable --log-level=info (or debug) by default and specify a log file so that apport can easily grab this data.
The log file will be overwritten each time, so space should not be a problem.
Cached Data
Checkbox already stores its report file (submission.xml) when run. This should be supplemented with cached system and submission IDs for apport (and other systems) to access, along with the launchpad id used to submit data.
This is linked to bugs 363549 & 379393 which deal with using FreeDesktop XDG base directory specifications and exposing the system/submission IDs to Apport respectively.
Use Cases
- Bob is reporting a bug using apport. Now that launchpadlib has the ability to link hw profiles to bug reports, it would be great if apport could parse the system for the stored system/submission id and use that to automatically link the bug report to the hw profile. Bob's bug report will then be more informative for developers looking to debug the issue.
Implementation Details
Two plaintext files are stored in $checkbox_data - system and submission which contain the system and submission IDs respectively.
Bug Filing
The Checkbox report will be extended to provide the facility for filing bugs via apport for all tests. Bugs can be filed directly on a package, when known, or against general functionality using apport's new symptom functionality.
Use Cases
- Paul's network card is not detected by Checkbox, so he is prompted to file a bug against Checkbox for this
- When Sarah runs the sound card test, it fails to work so she is prompted to file a bug against (ALSA/Pulse Audio)
Implementation Details
Each test will have a "Report Bug" link next to it in the report, which when activated will trigger Apport with either a package or symptom provided by the test.
This will require a "source" attribute to be provided in test definitions.
If no "source" is provided, ubuntu-bug will be used instead.
Checkbox QA package
A new package - checkbox-qa will be created in universe as a repository for extended tests and plugins above and beyond the base tests currently provided. The emphasis on tests in checkbox will be on end users (see "Terminology" below), while checkbox-qa will be technical and detailed tests.
Using this package as a catch-all for additional suites and plugins will make life easier for the various teams wanting to integrate their work with Checkbox.
This will also allow us to stabilise the Checkbox core packages, while rapidly expanding (and prototyping) a wider range of tests.
checkbox-qa-mago
Since a number of tests in the checkbox-qa package may depend on Mago, and it's not clear whether Mago will be packaged for Karmic, we should either:
Put checkbox-qa in a PPA along with Mago until the latter is available in the archives
Split the package into checkbox-qa (non-Mago tests, universe) and checkbox-qa-mago (Mago-only tests, PPA)
Use Cases
- Eitan wants to distribute his desktop tests (implemented through Mago) to the QA community
- Chris has written a webcam test that he wants to make available
Implementation Details
Creation of new package checkbox-qa in universe
Migration of some suites and plugins from checkbox-compatibility and checkbox-certification to the new package
Added dependency to -compatibility and -certification on the new package
- Integration of OEM and Security tests into new package
- Integration of desktop tests (Mago) when complete
Open Questions
- Should the suspend-resume test be moved to the new package?
Test Progress
Checkbox should display the total number of tests, and the current position.
Use Cases
- While running System Testing, Sheila wants to know how many tests there are so she can decide whether she wants to complete the full set of tests now
Implementation Details
The number of tests and current position will be displayed using natural language: "Test 3 of 8". This string will be translated as part of the usual translation process.
A status bar should be added to the Checkbox GTK interface, that displays the test progess.
The Checkbox CLI interface will display the test progress as a separator between each test.
Test Output
Checkbox should be able to display the output of tests. This will be useful for both debugging and long running tests.
Bug 333916 corresponds to this enhancement.
Use Cases
- While running a suite of Mago tests, Eitan wants to see the progress of the tests
- Mike cannot get his Internet test to pass, even though he knows he is connected. He analyses the test output to find the fault
Implementation Details
A progress bar dialogue should be displayed for each test that includes:
- The test name
- The test position (see above)
- A progress bar
- A "Show Test Details" button (changes to "Hide Test Details" when activated)
- A (hidden by default) terminal style pane for test output
Test output is automatically echoed/copied/piped to the output pane.
When the "Show/Hide" button is clicked, the output pane is displayed, and the button text changed.
The state of the "Show/Hide" button is remembered between tests, and the progress dialogue configured accordingly for each test.
What should the behaviour be if the test fails?
Unsure of how to address this for the CLI interface.
Local Usage
Although we want to encourage people to submit their data to Launchpad, it should be made more obvious that the report will be saved locally on the submission screen. The report will be displayed to the user who will then be given the option to save it and/or submit it to Launchpad.
When using checkbox-extras, local storage will be used. In the future results from the '-qa' package will be submitted to the QA Results Tracking System (QARTS).
A related bug is 305111.
Use Cases
Need some specific stories.
Implementation Details
- Separate the report summary/details from the Launchpad exchange plugin
- Improve the report summary/details
Make the Launchpad exchange optional in the -qa package
Terminology
The tests in the base package should avoid technical terminology where possible, instead presenting a user friendly alternative. E.g. "We have detected a wired network controller." vs "We have detected a Broadcom Foo 9000.".
This will require re-writing of the tests and/or their detection scripts.
Use Cases
- Sam uses System Testing to confirm that his network connection is working. He does not care about the technical details of his network adapter.
Implementation Details
Rewrite existing tests (in checkbox) to use natural language and minimise technical details or terms
Add expanded tests (in checkbox-qa) that duplicate the functionality of the re-written tests but retain the technical aspects