Ubuntu Wiki

• Launchpad Entry: karmic-qa-daily-desktop-testing

• Created:

• Contributors:

• Packages affected:

Summary

Add desktop tests to the daily hardware testing schedule on laptops and netbooks.

Rationale

The automated desktop testing framework and suites allows us to repeat a large volume of UI tests on a regular bases. Integrating these tests into the existing checkbox-based hardware certification testing setup will allow us to take advantage of it's schedule and results reporting system. Ubuntu's quality will greatly increase if we regularly run automated desktop tests.

Use Cases/Requirements

• Run all automated desktop test suites every day on a daily (installed) image.
• Run all automated desktop test suites every day on live CD.

Design

• The implementation will rely on Checkbox.
• It will use the certification website for reporting test results.
• There will be a method of tagging certain tests as hardware dependent, this will assure the tests are run on relevant target machines, and not on incompatible hardware or virtualized targets.
• The solution will be usable on both physical and virtual hardware, and will take advantages of the strengths of both:
  • PXE boot on hardware.
  • ISO loopback on VMs.
  • Snapshot mode it VMs.
  • Quick ghosting.

Implementation

• The checkbox certification plugin set will be expanded to include a Ubuntu Desktop Testing (UDT) plugin that will execute the whole gamut of included automated test suites.
• A Ubiquity suite will be authored for automating install tests.
• Parallelizing test execution in a cloud will be investigated, and perhaps implemented.
• The image will remain pristine as much as possible, the test harness will be executed from a working directory with all dependencies being local.
• A boot parameter will be required to run tests directly on a Live CD.

Test/Demo Plan

The tests will be tested once the tests are run on a regular testing schedule, and the test results will be displayed in a human-readable fashion on the current hardware test website.

Unresolved issues

The Live CD needs a boot option for executing a command on startup.

Future work

• Parallelize test suites across multiple targets to reduce test duration, perhaps with cloud-like elasticity.
• Run Ubiquity installation from a live CD.

Action Items

• determine 1-2 duplicate machines that can be removed from the PPA pool and used as permanent VM hosts (Ronald)
  • coordinate with Eitan
• talk to installer team and find out best way to get LiveCD tests to execute (Marc)

CategorySpec