Ubuntu Wiki

• Launchpad Entry: karmic-qa-checkbox-expand-test-coverage

• Created: UDS Karmic

• Contributors: Ronald, Marc, Dave

• Packages affected: Checkbox

Summary

Add further test coverage to checkbox, including Webcam, Audio and Printing.

Release Note

More system tests, both manual and automatic, have been added. This enables Ubuntu to be tested on a wider variety of hardware and focus development on hardware that does not work properly.

Rationale

In order to provide a high-quality product, we must test as much of it as possible on as much different hardware as possible.

User stories

• Fred has some unusual hardware that is not supported under the latest beta release of Ubuntu. He would like to be able to test what capabilities are supported and file bugs against those that are not.
• Certification testing will take advasntage of a richer pool of tests for more in-depth testing.

Design

Tests will be written or imported and added to test suites within Checkbox. These will in turn be distributed over several packages:

• checkbox - basic tests installed by default on all systems (main)
• checkbox-extras - extended test coverage for in-depth testing by the Ubuntu community (universe)
• checkbox-compatibility - used by hardware partners to validate systems (PPA)
• checkbox-certification - run in the certification labs (PPA)

In released versions of Ubuntu, Checkbox will default to a basic set of tests as only the base checkbox package is installed. Any user can install checkbox-extras for further test coverage. Hardware partners will also be provided with checkbox-compatibility.

The checkbox-compatibility and -certification packages will not depend on -extras. In fact they will be designed to conflict and there will be significant overlap in coverage. A test may first appear in the universe package where it is tested before graduating to the commercial packages while other cases may start private and later be released in -extras.

Test Categories

The following broad categories should have tests developed:

• audio
• video
• networking
• input devices
• hibernate
• suspend
• other
  • miscellaneous hardware and software

Implementation

Additional tests

The tests below will be added to Checkbox as tests for Karmic. Tests marked as "manual" have at least some manual requirements, even if the test is partially automatic (e.g. confirmation of results or connection of an external device). Tests in the checkbox package will be shipped in the default Checkbox tests, while those in the checkbox-extras package will not be installed by default, but can be added by interested parties.

[1]: Basic audio playback and recording are already implemented as simple Checkbox tests and not represented in this chart. All audio tests here are specifically in-depth tests and extend what is already present.

[2]: This test is conducted automatically but requires hardware setup ahead of time. The default audio output will be directly connected to the default audio input for testing.

[3]: Portions of the test can be run automatically without requiring user input, which will be the default for automated tests. Extended tests require manual intervention to perform actions such as opening/closing the lid and removing the power cord. This full set of tests will be the default when Checkbox is run interactively.

Additional registry information

In addition to the expanded test coverage outlined above, Checkbox will retrieve the DSTD information from the system BIOS and return it as part of the hardware information. This information can be retrieved via a script that the kernel team has written. This script will be added to the Checkbox registry plugin by the Checkbox team.

Import ecurity team tests

Which?, How?

Integration of OEM Tests

The OEM team has a number of tests already written or being written that will be integrated as a part of this process. These tests will be added to the checkbox or checkbox-extras packages as outlined above.

Automatic Tests

Wherever possible, tests should be automated. Automated tests that do not require an inordinately long time to run or are experimental/potentially damaging to the user's system can be included in the default checkbox package.

Manual Tests

Many extended tests will be written as manual tests. Since these user interaction they will not be added to the standard package but instead to -extras and PPAs.

Test/Demo Plan

Additional tests will be initially added to the development version of Checkbox and tested by members of the QA team and by interested community members who install the development release of Ubuntu. Feedback from these tests will be used in part to determine which suites of tests should be enabled by default and which should be in the extended selections.

Unresolved issues

Certain tests may be possible to automate but will be initially created as manual tests. Future specifications will deal specifically with automation of extended testing. Tests that were once 'extended' manual tests that have been automated may become part of the 'standard' suite of default Checkbox tests, as they will have little or no user impact.

BoF agenda and discussion

Make sure all useful bits from here have made it into the spec:

What can be added to Checkbox?

• ability to grab DSTD (from BIOS) and disassemble it and check the code
  • apparently there are scripts to do this
  • it can tell us if the BIOS is fundamentally broken
    • it sounds like this would be better to include as information in the hardware submission
• OEM team has a directive to add cases to checkbox
  • already added compiz compatibility, suspend/resume stress test, camera tests
• audio test -- linking input and output, generating and testing for waveforms
• webcam
• printer
• modem
• 3g
• hotkeys/media keys
• video out
• cardreaders
• usb/firewire
• tv tuners
• gps
• hibernate
• suspend
• power drain during suspend (long-term suspend test)

CategorySpec