SharingTestingInfrastructure
Launchpad Entry: foundations-m-upgrade-and-install-testing
Created:
Contributors: EvanDandrea
Packages affected: ubiquity
Summary
Members of the foundations team are creating infrastructure to test critical software in the distribution. This work should be continued to expand coverage. Steps should also be taken to share overlapping infrastructure and centralize publishing of data where possible.
Release Note
- The installer is now built and testing under a public-facing continuous integration system.
Rationale
- Bugs in the installer have nearly caused us to miss the targeted release date for Ubuntu on more than one occasion. In most cases this was the result of trying to fix one critical or opportunistic bug, only to create an equal or greater problem.
User stories
Assumptions
We will be able to run Hudson on Canonical-owned, Internet-facing hardware.
Design
- A public facing instance of Hudson will collect test results from a set of slave machines, both physical and virtual, or possibly from a private Hudson that speaks to datacenter-located physical hardware slaves.
- This public facing instance will at a minimum run tests for the installer, but may also collect data from existing sources such as the ISO testing tracker and Michael's upgrade tests, or provide a means for the ISO testing tracker to source automated test data (via Hudson's remote API).
Implementation
Installer testing
A Hudson instance will be configured with a set of jobs that will do the following:
Build ubiquity on a new bzr commit (or possibly on a tag) and run its in-tree tests. The tests will be written to JUnit XML using python-junitxml and published in Hudson.
If the ubiquity build succeeds, an ISO will be remastered with the resulting debs and the dependencies for Sikuli (openjdk-6-jre, libcv4, libcvaux4, and libhighgui4). These debs will be placed in pool/ rather than the squashfs to prevent contamination of the install tests.
- The ISO will be loopmounted and exported over NFS. A running PXE server will be restarted to point to the new nfsroot.
- A Hudson job will then ask all slaves to wipe their MBR and reboot once they're done with their current task. It will then schedule live environment installation tests for a group that contains all of the slaves.
Upon rebooting, the slaves will PXE boot into the aforementioned live image and start the Hudson slave via JNLP. The pending live environment installation test job will then be delivered to each slave, which will clean the build tree, pull down sikuli and a set of tests, and then run them. The results will be written to JUnit XML using python-junitxml and published in Hudson.
Test/Demo Plan
Unresolved issues
BoF agenda and discussion
Notes from OEM and Colin Watson: * More stress testing before release * Improving our testing procedures * Using more automated tests * Further brainstorming Some other automatic tests we have and that may need improvements: Package install testing (working, but needs features kill auto-kill for hanging installs): http://people.canonical.com/~mvo/automatic-upgrade-testing/auto-install-tester/ Automatic upgrade testing (no gui): http://people.canonical.com/~mvo/automatic-upgrade-testing/current/ Notes from Evan: http://people.canonical.com/~scott/daily-installer/
Upgrade and Install Testing =========================== * upgrade tests is catching a fair amount of bugs * upgrade testing should also test universe (it does not currently because of scaling issues) * testing that the upgraded system "works" is fairly limited at the moment * some (relatively small number of) packages fail to install even on a clean system; these are easy to fix * does anyone still use the weather report? steve and leann probably still do * testing in the cloud? * release upgrader has an EC2 mode * ... but kvm tends to be higher-value, at least for desktop-oriented tests * feasibility of EC2 testing for install? * use EC2 for the automatic install/remove test * try to create autotest profile for ubiquity (via IS) - machine name is pommerac * ACTION: look at hudson or any other continuous integration testing frameworks * use a single instance of hudson to publish all of our tests * need to be able to easily plug in new machines to existing test frameworks * good candidate for sitting between all of our systems and report back? possibly * can it store test output? * talk to Robert Collins * ACTION: look into handover of Soren's testing system to QA (Carlos)? * ACTION: mdz to work with QA to task someone with overseeing our test process * no one has been guiding this effort from a larger, architectural perspective and this needs to be done * ACTION: figure out what to do about failed packages * list the systems for which we already have a central testing framework in place * python testing * pyflakes.vim is made of awesome, for those who don't already use it * pychecker has a lot of false positives. pyflakes is apparently much better and maintained by a Canonical employee ACTION: james_w and cjwatson * Launchpad PPA with permissions matching Ubuntu archive permissions * Bot monitors this PPA and runs tests on packages in it, and copies them into the primary archive if they work * piuparts is more configurable now and we should look into it again to see if it can be used to provide a good set of tests without too much false positivies