MobileBringupTesting

Summary

New hardware for mobile projects should go through a uniform test procedure during bringup. At this time, a test summary of what works, what doesn't, and any workarounds or extra pieces required for functionality should be documented. This should also be useful for helping others to get the hardware up and running, and know what to expect.

Release Note

No user visible change in the distribution.

Rationale

There is sometimes a limited supply of hardware available for testing and development on new devices. This can limit the amount of testing that can happen, and delay it until too late in the cycle. Having a documented procedure for a baseline of what works and what doesn't can allow anyone with the hardware to test it as soon as anyone has the hardware available. Testing can continue from there with a knowledge of which pieces are already known to be broken (with bug numbers available) and which worked at one time, but may regress in later builds.

User stories

Peter just got a brand new XYZ development board to do bringup on. Peter runs through a short series of tests, files bugs for things that do not work, and documents the status of each item, with links to bugs, and any workarounds he used to get the system up and running using a wiki template.

Susan received her XYZ development board a few months later to test on. She is able to get it installed an running quickly since Peter already documented any needed workarounds on the wiki. She re-runs the tests and updates the bugs and wiki with her results

Design

When complete, we should have a bzr branch of checkbox that can be pulled to a new, installed machine. There will be a set of tests defined as a checkbox test suite that will run through the set of tests we defined here and produce a wiki formatted text file as output. This can be directly pasted onto the wiki for the machine being tested, and details can be filled in as needed.

Implementation

  • identify or create tests in checkbox for the things we want to test
  • write a launch timer script that will launch a gui app and give the time it took to start
  • write a script that will pull arm-specific system information such as the uboot or redboot version
  • create a plugin for checkbox to output results in wiki format

Test details

  • All supported video output (and which resolutions are supported)
    • already provided, though compiz check may be unnecessary at this time
  • Audio output and input
    • already in checkbox, some tests for USB audio would be interesting also but require additional hardware
    • Tobin mentioned that he may try to expand on these some so that we can test the various layers of audio support
  • onboard networking (wired or wireless)
    • tests in checkbox for wired networking, but should also include wireless if there is a wireless device
  • USB Storage
    • needed - simple test to ask user to insert usb stick, read/write some files
  • USB network
    • needed - (if hardware available), should work similarly to the other networking tests
  • volume and muting
    • needed - should have a test button that plays a sound before/after so that the user can tell if the volume has changed or if the muting has worked
  • brightness
    • needed - sufficient to ask the user to check this manually, but the test should also ask the user to do it with keyboard buttons if they are available
  • SD card slots
    • needed - similar to USB storage
  • SATA
    • needed - not a good standard way of doing this. If the system is installed to a sata device then that is probably sufficient, otherwise we are basically asking the user here "were you able, either on this install or some previous install test, able to install and run the system on a sata device"
  • Suspend/resume
    • there is already a script to support this type of testing in checkbox, need to try it and see if it works, and write the checkbox test to just use it. smoke testing is sufficient for this phase
  • hibernate
    • I don't think the suspend/resume script currently supports this, but this part could be either scripted to do it for the user, or ask the user to do it manually.
  • bootchart
    • image conversion currently broken on arm, so will need to gather the raw data. Would be good if there were a way to just extract something useful from it, but will have to take a look and see. This one is probably of lower priority compared to the others.

Code Changes

  1. a script to determine uboot/redboot version and log it with the rest of the system information
  2. changes to existing checkbox tests and adding new tests as mentioned above.
  3. a plugin will be added for checkbox to allow wiki formatted export of test results

Unresolved issues

Some of the "tests" in this case will require that the user do some checking or setup before testing. For instance, multiple video outputs, sata, etc, may need to be manually identified by the tester and noted in the test comments.

BoF agenda and discussion

= Establish test procedures and documentation for new hardware =
mobile-lucid-bringup-testing
https://wiki.ubuntu.com/Specs/MobileBringupTesting

== Testing Tasks ==
A full pass of the hardware device on the device. This should not be an exhaustive list of tests, but more of a general "smoke test" of available hardware interfaces.

 * all supported video output (and which resolutions are supported)
 • Audio output alsa and pulseaudio
 • audio input
 • onboard networking (wired or wireless)
 • USB storage device
 • USB network device
 • volume and muting
 • brightness
 • SD card slots
 • sata
 • Suspend/resume
 • hibernate
 • power off (should actually power off the machine)
 • reboot
 • any other devices or external ports the board happens to have (please specify)
 • retains date/time across power off and disconnected power?
 • bootchart? does this even work?  If nobody knows, need an action item to go find out.
 • Also need to do time how long it takes to bring up a few common, but large apps (i.e. web browser, open office).
 • cpuidle (power management testing)

gather system information (checkbox should already handle this)
create trimmed down suspend/resume/hibernate test, or use the one the kernel team already has (though there is another spec for more robust suspend/resume/hibernate testing, so a shorter test here is appropriate)
plugin for checkbox is needed to convert output to wiki format (cr3/plars)
Figure out a way to determine redboot/uboot version and log it (ogra)
look into providing better audio tests (gruemaster) - we discussed having a audio tested in stages to better determine the source of the problem if problems are found


CategorySpec

Specs/MobileBringupTesting (last edited 2009-11-25 02:18:21 by g229069207)