• Launchpad Entry: server-lucid-automated-kvm-testing

• Created: 2009-11-23

• Contributors: DustinKirkland

• Packages affected: qemu-kvm, libvirt

Summary

We should define a stack of functionality and performance tests and run them for every new revision of qemu-kvm and libvirt, and various configurations.

Release Note

Ubuntu 10.04's virtualization infrastrucure has undergone regular, automated testing.

Rationale

KVM is the Ubuntu's virtualization hypervisor, and Libvirt is the abstraction layer by which Eucalyptus communicates with KVM. These fundamental technologies are the foundation upon which we build the Ubuntu Enterprise Cloud.

We can, and should automate the testing of KVM and Libvirt on a regular basis.

User stories

• As an administrator of a local installation of UEC, I wants to ensure the stability her users' instances, which run in KVM virtual machines. With Canonical/Ubuntu having regularly and heavily tested its virtualization stack, I can rest assured that my KVM/Libvirt virtual machines will be reliable and high-performance
• As a developer of UEC, I wants to ensure that the KVM and Libvirt virtualization stack are stable and bug free. I can trust the virtualization foundation and focus my efforts on debugging UEC.
• As a system administrator interested in migrating my infrastructure to virtualization-based solutions, I chose Ubuntu/KVM/Libvirt based on the extensive testing performed by Canonical and Ubuntu in the Lucid LTS cycle.

Assumptions

• Canonical IS will provide some VT-capable hardware which can run these automated tests.
• Canonical QA will instrument checkbox or some other testing framework to enhance the test coverage, automatically trigger these tests, review the results, and report bugs accordingly.
• Canonical Server Team will respond and fix bugs found/filed by QA.

Design/Implementation

Details in the sections below.

How to Test

• Autotest, KVM-Autotest

  • Wide support with Google, Red Hat, IBM, etc.
  • Use Step Files for different distros.
  • Currently use automated scripts to test for certain end products.
  • Ubuntu community should build Step Files, and get them committed to Autotest.
  • Autotest currently doesn't have a large number of fucntion tests.
    • Ubuntu could greatly expand these tests.

What to Test

• Functionality, Performance, Stability, Regression
• QEMU-KVM

  • Already tested by KVM-Autotest

  • We need to integrate these tests into Canonical's automated testing infrastructure (checkbox?)
  • Could also enhance the functional testing coverage
• Libvirt
  • Do libvirt testing as well as KVM testing, as UEC users are affected by both of these
  • Add libvirt tests to Autotest.
  • libvirt-sim does some tests with Autotests.
  • Security team has some libvirt tests
• Guest OSes
  • Linux
    • Ubuntu 8.04 LTS, 10.04 LTS, Desktop, Server
    • Debian 5.0
    • Fedora 12
    • CentOS 5
    • OpenSUSE 11.2
    • amd64 and i386
    • desktop and server
  • Windows
    • Windows 2003, XP, Vista, 7
  • Ensure that these guest OSes
    • Install
    • Boot
    • Have network connectivity
    • Sound (less important)
    • USB passthrough (less important)

Procedure

• Build a collection of images that we care about testing
  • For Ubuntu, we could perhaps use daily UEC images (?)
  • If Windows is interesting, can use cygwin to get a bash shell
• Guests attach a bash shell to the serial port
• Test scripts interact with the shell
• Run KVM-Autotest's functional tests
  • May need to enhance these (working with the upstream project) to ensure we cover the functionality that's important to us
    • e.g. dynamic block storage attach is something that we need for UEC, but it's not currently tested by upstream
  • Every test has a timeout
• Do performance testing by timing certain processes (using existing checkbox performance tests)
  • Network throughput
  • Disk throughput
  • CPU intensive calculations
  • Memory intensive calculations
  • Track results over time to identify if there is a regression
• Integrate KVM-Autotest framework into hardware certification testing to "certify" KVM as a support hardware platform for Ubuntu releases
  • Treat as a native hardware platform.
  • Currently VMware is treated in this fashion.

When to Test

• Alpha1, Alpha2, Alpha3, Beta1, Beta2, RC
• At each upload of qemu-kvm, libvirt
• Perhaps one day link this and test against the daily-builds

Upstream Contribution

• Should submit "Step Files" to upstream kvm-autotest for supported Ubuntu releases (i386 and amd64)
  • These are more or less "preseed" files
• Additional upstream tests

Risks

The Ubuntu QA Server position is not filled in time to address all the blueprints in Lucid

Category: Human resources
Probability: 80%
Risk threat level: High

Mitigation plan

Mitigation Strategy:
Use the KVM related work items in the automated server testing (https://blueprints.edge.launchpad.net/ubuntu/+spec/qa-lucid-automated-server-testing) to address a minimal coverage for KVM testing.

Contingency:

1. Ubuntu QA Manager: Postpone any work items that are not required to address a minimal coverage of KVM

2. Ara Pulido: Use Automated Server Testing KVM items to address a minimal coverage.

3. Ubuntu QA Manager: Decide if it is worth the effort required to create a blueprint for the M release to cover the rest of the work items

Test/Demo Plan

• Once the automation is in place, execute the automated tests whenever a new qemu-kvm or libvirt package is uploaded to the archive.
• Provide the results in a web hosted manner.
• Act upon the results by confirming any issues found, and filing appropriate bugs.

Unresolved issues

• Need a named QA resource dedicated to this project
• Need agreement on hardware in the Canonical lab where these tests can run

Discussion

---

CategorySpec