##(see the SpecSpec for an explanation)

 * '''Launchpad Entry''': [[https://blueprints.edge.launchpad.net/checkbox/+spec/karmic-qa-checkbox-expand-test-coverage|karmic-qa-checkbox-expand-test-coverage]]
 * '''Created''': UDS Karmic
 * '''Contributors''': Ronald, Marc, Dave
 * '''Packages affected''': Checkbox

== Summary ==

Add further test coverage to checkbox, including Webcam, Audio and Printing.

== Release Note ==

More system tests, both manual and automatic, have been added.  This enables Ubuntu to be tested on a wider variety of hardware and focus development on hardware that does not work properly.

== Rationale ==

In order to provide a high-quality product, we must test as much of it as possible on as much different hardware as possible.

== User stories ==

 * Fred has some unusual hardware that is not supported under the latest beta release of Ubuntu.  He would like to be able to test what capabilities are supported and file bugs against those that are not.
 * Certification testing will take advasntage of a richer pool of tests for more in-depth testing.

== Design ==

Tests will be written or imported and added to test suites within Checkbox. These will in turn be distributed over several packages:

 * checkbox - basic tests installed by default on all systems (main)
 * checkbox-extras - extended test coverage for in-depth testing by the Ubuntu community (universe)
 * checkbox-compatibility - used by hardware partners to validate systems (PPA)
 * checkbox-certification - run in the certification labs (PPA)

In released versions of Ubuntu, Checkbox will default to a basic set of tests as only the base checkbox package is installed. Any user can install checkbox-extras for further test coverage. Hardware partners will also be provided with checkbox-compatibility.

The checkbox-compatibility and -certification packages will not depend on -extras. In fact they will be designed to conflict and there will be significant overlap in coverage. A test may first appear in the universe package where it is tested before graduating to the commercial packages while other cases may start private and later be released in -extras.

=== Test Categories ===

The following broad categories should have tests developed:
 * audio
 * video
 * networking
 * input devices
 * hibernate
 * suspend
 * other
  * miscellaneous hardware and software


== Implementation ==

=== Additional tests ===

The tests below will be added to Checkbox as tests for Karmic.  Tests marked as "manual" have at least some manual requirements, even if the test is partially automatic (e.g. confirmation of results or connection of an external device).  Tests in the ''checkbox'' package will be shipped in the default Checkbox tests, while those in the ''checkbox-extras'' package will not be installed by default, but can be added by interested parties.

|| Test || Description || Manual/Automatic || Package || Implementation Details || Script Exists ||
||<-6>'''Audio'''[1] ||
|| audio-loopback || Test basic audio playback/recording || automatic[2] || checkbox-extras || Implemented by Checkbox team; see [[https://blueprints.launchpad.net/ubuntu/+spec/karmic-qa-extended-audio-testing|Extend Audio Testing]] specification || ||
|| audio-extended || Test additional audio hardware and jacks || manual || checkbox-extras || Implemented by OEM team and in the [[https://blueprints.launchpad.net/ubuntu/+spec/karmic-qa-extended-audio-testing|Extend Audio Testing]] specification || ||
||<-6>'''Video''' ||
|| compiz-check || Test for compiz functionality || automatic || checkbox || Implemented by OEM team || ||
|| x-check || Test X server and modes || automatic(?) <!> || checkbox-extras || <!> Bryce? || ||
|| display-external || Test external display (VGA/DVI) || manual || checkbox-extras || Implemented by OEM team || ||
|| display-hdmi || Test HDMI video/audio output || manual || checkbox-extras || Implemented by OEM team || ||
|| dvd-playback || Test DVD playback || manual || checkbox-extras || Implemented by OEM team || ||
||<-6>'''Webcam''' ||
|| camera-video || Test webcam video capture/playback ||  manual || checkbox || Implemented by OEM team || ||
|| camera-still || Test webcam still image capture || manual || checkbox || Implemented by OEM team || ||
|| camera || Camera detection || automatic || checkbox || Implemented by OEM team || ||
||<-6>'''Networking''' ||
|| wireless-bluetooth || Test wireless networking via Bluetooth || manual || checkbox-extras || Implemented by OEM team || ||
|| wireless-wifi || Detect wifi hardware and test connectivity || automatic || checkbox || Implemented by OEM team || ||
|| wireless-wifi-quality || Analyse wifi signal/noise quality || automatic || checkbox-extras || Implemented by OEM team || ||
|| modem || Detect modem || automatic || checkbox || Implemented by OEM team || ||
||<-6>'''Suspend/Resume''' ||
|| suspend-lid || Test suspend/resume via lid switch || manual || checkbox || Implemented by OEM team || ||
|| suspend || Test suspend/resume via software || automatic || checkbox || Implemented by OEM team || ||
|| hibernate || Test hibernate/resume. || automatic || checkbox || Implemented by OEM team || ||
|| suspend-stress || Stress-test suspend/resume || automatic/manual[3] || checkbox-extras || Existing kernel team script will be integrated by Checkbox team || ||
|| fs-performance || Test filesystem performance || automatic || checkbox-extras || Implemented by OEM team || ||
||<-6>'''Input Devices''' ||
|| fingerprint || Test fingerprint reader || manual || checkbox-extras || Implemented by OEM team || ||
|| keyboard-hotkeys || Test media keys/hotkeys || manual || checkbox || Implemented by OEM team || ||
||<-6>'''Other''' ||
|| ldtp || Linux Desktop Testing Project tests || automatic(?) <!> || checkbox-extras || <!> || ||
|| security || Kernel security scripts || automatic || checkbox-extras || <!> || ||
|| firewire || Test Firewire support || manual || checkbox-extras || Implemented by OEM team || ||
|| cardreader || Test media (SD, CF, etc.) card readers || manual || checkbox || Implemented by OEM team || ||

[1]: Basic audio playback and recording are already implemented as simple Checkbox tests and not represented in this chart.  All audio tests here are specifically in-depth tests and extend what is already present.

[2]: This test is conducted automatically but requires hardware setup ahead of time.  The default audio output will be directly connected to the default audio input for testing.

[3]: Portions of the test can be run automatically without requiring user input, which will be the default for automated tests.  Extended tests require manual intervention to perform actions such as opening/closing the lid and removing the power cord.  This full set of tests will be the default when Checkbox is run interactively.

=== Additional registry information ===

In addition to the expanded test coverage outlined above, Checkbox will retrieve the DSTD information from the system BIOS and return it as part of the hardware information.  This information can be retrieved via a script that the kernel team has written.  This script will be added to the Checkbox registry plugin by the Checkbox team.

=== Import ecurity team tests ===

<!> '''Which?, How?'''

=== Integration of OEM Tests ===

The OEM team has a number of tests already written or being written that will be integrated as a part of this process.  These tests will be added to the ''checkbox'' or ''checkbox-extras'' packages as outlined above.

=== Automatic Tests ===

Wherever possible, tests should be automated.  Automated tests that do not require an inordinately long time to run or are experimental/potentially damaging to the user's system can be included in the default checkbox package.

=== Manual Tests ===

Many extended tests will be written as manual tests.  Since these user interaction they will not be added to the standard package but instead to -extras and PPAs.

== Test/Demo Plan ==

Additional tests will be initially added to the development version of Checkbox and tested by members of the QA team and by interested community members who install the development release of Ubuntu.  Feedback from these tests will be used in part to determine which suites of tests should be enabled by default and which should be in the extended selections.

== Unresolved issues ==

Certain tests may be possible to automate but will be initially created as manual tests.  Future specifications will deal specifically with automation of extended testing.  Tests that were once 'extended' manual tests that have been automated may become part of the 'standard' suite of default Checkbox tests, as they will have little or no user impact. 

== BoF agenda and discussion ==

'''Make sure all useful bits from here have made it into the spec:'''

What can be added to Checkbox?
 * ability to grab DSTD (from BIOS) and disassemble it and check the code
  * apparently there are scripts to do this
  * it can tell us if the BIOS is fundamentally broken
   * it sounds like this would be better to include as information in the hardware submission
 * OEM team has a directive to add cases to checkbox
  * already added compiz compatibility, suspend/resume stress test, camera tests
 * audio test -- linking input and output, generating and testing for waveforms
 * webcam
 * printer
 * modem
 * 3g
 * hotkeys/media keys
 * video out
 * cardreaders
 * usb/firewire
 * tv tuners
 * gps
 * hibernate
 * suspend
 * power drain during suspend (long-term suspend test)



----
CategorySpec