Ubuntu Wiki

Please check the status of this specification in Launchpad before editing it. If it is Approved, contact the Assignee or another knowledgeable person before making changes.

WORK IN PROGRESS

• Launchpad Entry: ubiquity-vmware-automation

• Packages affected: vmware-ubuntu

Summary

We will develop a test framework for Ubiquity, the desktop CD installer, that will drive the installer via preseed files and ATK inside VMWare Server.

Release Note

A unit test framework for the desktop CD installer has been developed and deployed. Tests for successful installs now run with each new daily build of the Ubuntu desktop CD.

Rationale

Testing Ubiquity for installation success in a number of different configurations is a tedious, but necessary, process. This needs to be done by the Ubiquity developers constantly, and by the QA team at every milestone for every supported architecture. This process can be automated and run at a much greater frequency than is currently capable. The resulting data can be fed into developer-weather-report.

Use Cases

• Evan is an Ubiquity developer. He releases a new version of Ubiquity and wants to test it against a wide array of different hard drive and device configurations to spot new bugs and regressions.
• Lars is a member of the QA Team and needs to test the installer on all of the CD images for every milestone. As this process mostly involves clicking the next button and filling in forms with basic information, it can be automated and his time can be better applied elsewhere.

Assumptions

• VMWare tools will be included on the desktop CD seeds.
• A system built to meet the minimum installation specifications with VMWare Server installed will be available.

Design

A number of tests will be developed. A test will merely be a specification of the hardware and disk configuration that need to be present on the virtual machine and a matching preseed file or ATK script, in future versions, written in dogtail or a similar meta-language. A test for successful installation without an active network connection could include a blank 3GB vmdk disk and no network card, whereas a test for succesful install on a system where Windows Vista is already present could include a 10GB disk, with a 7GB Vista partition.

Propreitary operating systems, such as Windows Server 2003 and Windows Vista, can be obtained from the vendor for the purpose of virtual machine installation free for a timed trial, and then converted into VMWare format using qemu-img. If the license for these virtual machines prevents us from implementing this specification legally, skeleton operating systems can be constructed using free tools.

A script will be developed to check for any new desktop CDs that have not been checked against the list of tests, download them, and run them one by one against every test one by one. Normally there should be only a single CD to download as this script will run daily.

A script will be developed to drive the individual tests. It will create a VMWare VM, use Marc Tardif's vmware-ubuntu script to remaster the day's daily ISO with the preseed from the current test in the case of preseed testing, boot from the CD in the VM, and then wait for the VM's state to either change to suspended or shutdown by querying vmware-cmd and sleeping in a loop. If the virtual machine state changes to shutdown, it is deleted and the test result page, a public facing website, will be updated to reflect test success for the given image, test, and date. If the virtual machine state changes to suspended, it is moved to a publically accessible location and the tests results page will be updated to reflect test failure for the given image, test, and date, with a link to download the suspended image for further investigation.

The preseed file for each test will provide a ubiquity/success_command preseed to shutdown the virtual machine if the install completes succesfully and a ubiquity/failure_command preseed to suspend the virtual machine if the install fails. This will need to be changed to account for upcoming changes to ubiquity that split apart install failure and automation failure into two seperate preseeds. As the install could fail without propigating the failure up to failure_command, d-i/early_command will also be preseeded to fork a reasonably large timer and following suspend command.

Implementation

This section should describe a plan of action (the "how") to implement the changes discussed. Could include subsections like:

Test/Demo Plan

• Simple tests, such as running through an install using the full disk and ignoring migration-assistant will be developed to initially test this functionality.

Outstanding Issues

• Support for providing ATK-driven tests using Dogtail will need to be added once the preseed testing proves stable.
• Can we extract the log files from the installer on test failure and include them in the test report?

CategorySpec