MobileInstallTesting

Differences between revisions 3 and 4
⇤ ← Revision 3 as of 2009-11-19 16:03:41 →
Size: 5842
Editor: 63
Comment:
← Revision 4 as of 2009-11-23 19:22:30 → ⇥
Size: 5447
Editor: 75-27-138-126
Comment:
Deletions are marked like this. Additions are marked like this.
Line 28: Line 28:

You can have subsections that better describe specific parts of the issue.		 The idea we discussed in the session at UDS was to do the extra install testing after the milestone images are released. ISO tracker smoke testing will still operate as normal, but will be augmented by further install testing. For instance, testing on Beta1 will mean that the tests have been performed on a daily snapshot sometime between beta1 and beta2. A fake target will be created on iso tracker to facilitate testing of the images in this way. A smaller subset of the tests will be targetted for running on the pre-FINAL image.
Line 32: Line 31:
 * Create list of possible install options for arm
 * Create matrix of minimum tests required to test each option and each pair
 * Document install test scenarios on testcases.qa.ubuntu.com
 * Create virtual milestones for post-milestone testing and link to appropriate tests
 * perform testing at each milestone
Line 33: Line 37:
This section should describe a plan of action (the "how") to implement the changes discussed. Could include subsections like:

=== UI Changes ===

Should cover changes required to the UI, or specific UI that is required to implement this

=== Code Changes ===

Code changes should include an overview of what needs to change, and in some cases even the specific details.

=== Migration ===

Include:
 * data migration, if any
 * redirects from old URLs to new ones, if any
 * how users will be pointed to the new way of doing things, if necessary.

== Test/Demo Plan ==

It's important that we are able to test new features, and demonstrate them to users. Use this section to describe a short plan that anybody can follow that demonstrates the feature is working. This can then be used during testing, and to show off after release. Please add an entry to http://testcases.qa.ubuntu.com/Coverage/NewFeatures for tracking test coverage.

This need not be added or completed until the specification is nearing beta.

== Unresolved issues ==

This should highlight any issues that should be addressed in further specifications, and not problems with the specification itself; since any specification with problems cannot be approved.
Line 107: Line 85:
Possibilities for options to test during install:
Line 123: Line 101:
Network
* Connected/disconnected

Summary

The number of images to test is increasing rapidly, and some images require specific hardware which is not widely available in the community. These factors make it impractical to do exhaustive testing of install images. However, adequate coverage must still be acheived in order to ensure the quality of milestone images.

Release Note

No user visible change in the distribution.

Rationale

Ubuntu image installation is currently handled mainly through iso tracker, however, this is targeted mainly at smoketesting before a milestone. More comprehensive testing of install images is needed before the final release. Tests should be established that provide good coverage while limiting the number of tests required.

User stories

Lucy only has one XYZ development board to test with, and the candidates for the final image are now posted so she also has a limited amount of time. She wants to cover as many installation options as possible, but installs are time consuming. Using pairwise methods, she works out a plan that covers all of the options she wants, using only a few installs.

Assumptions

Most bugs are apparent with the combination of two or less factors being tested.

Design

The idea we discussed in the session at UDS was to do the extra install testing after the milestone images are released. ISO tracker smoke testing will still operate as normal, but will be augmented by further install testing. For instance, testing on Beta1 will mean that the tests have been performed on a daily snapshot sometime between beta1 and beta2. A fake target will be created on iso tracker to facilitate testing of the images in this way. A smaller subset of the tests will be targetted for running on the pre-FINAL image.

Implementation

  • Create list of possible install options for arm
  • Create matrix of minimum tests required to test each option and each pair
  • Document install test scenarios on testcases.qa.ubuntu.com
  • Create virtual milestones for post-milestone testing and link to appropriate tests
  • perform testing at each milestone

BoF agenda and discussion

= Efficient testing of install images =
https://wiki.ubuntu.com/Specs/MobileInstallTesting

* not mobile specific
* a lot of install testing towards the end of cycle
 * ad-hoc
 * some architectures like arm dont have a broad testing community
  * images take quite long to install - just one board/person
  * idea: limit the number of installs to perform
 * found a lot of install issues during the final release testing that were
   not really mobile specific.

* atm most testing is done throgh iso tracker
  -> goal: determine whether an image is good enough a milestone, but does
     stop right after the milestone is out
  -> problem: no real QA for images after the milestone is out
  -> problem: only covers default install, but no customizations
 * new approach pairwise (aka all-pairs) testing: minimize the number of tests to perform
   -> tools exist to generate the matrix to test
   -> question: how to reduce the matrix to something coverable in practice
      -> test higher risk combinations only
      -> example: be able to reduce test scenarios from 72 to 12 or even further
 * idea is to supplement this with a new testing approach
 * problem: testing hardware combinations
  * voluntarily submit hardware info (maybe using checkbox information) 
 * spread out testing of the pairs throughout the alpha cycle using the daily
   image
 * during release time just test the options PLUS verification of all the
   combinations with the pair elements involved in bugs found during previous
   testing -> see http://burtleburtle.net/bob/math/jenny.html
 * create a) fake milestones and b) fake products to a) allow testing of final milestone
   releases rather than milestone candidates and b) don't spam/distract community
   testers
 * is it a problem that test coverage is not balanced propertly (e.g. lots of users
   did test A ... but only a few do test B)? seems not to be the case
 * claim mechanism with timestamps

List of options to test (pairwise) - initially assembled from UNR install options
(might be slightly different for ARM images in lucid)

We can use pair wise testing between milestones:
 * Create fake milestones and products for those.
 * We have to avoid spamming the community members with new builds
 
Possibilities for options to test during install:
 Boot
• livecd + install
• install Ubuntu
OEM install
• on/off
Partitioning
• side-by-side
• guided
• manual
Login
• auto
• password
• password+ecryptfs home dir
Free software only
• on/off
Network
* Connected/disconnected

How can we know which hardware the community has:
 * The ISO tracker profile include a text field to specify the hardware
 * Using checkbox to send the hw

ACTION: get a better list of arm install tests/pairs
ACTION: checking and setting up virtual milestones and virtual products in iso tracker
ACTION: put a testing timeline/roadmap on wiki (two columns: a) scheduled/regular
        b) regression testcases etc.

CategorySpec

Specs/MobileInstallTesting (last edited 2009-12-16 22:59:26 by 75-27-138-126)