Ubuntu Wiki

• Launchpad Entry: other-p-qa-baremetal-testing

• Created:

• Contributors:

• Packages affected:

Summary

Test automation on bare-metal systems

Release Note

This specification is mainly to raise the awareness of the need to enable test automation on bare-metal systems (i.e. without VM support). Too many test developers focus on kvm for their testing needs.

Rationale

Too much of our test automation currently relies on KVM/Libvirt or custom system images, making it difficult to impossible to run on systems that do not support KVM (Arm, low-end NAS systems, etc) without a complete rewrite.

User stories

The Acme company wants to do some performance analysis testing on several different platforms to determine the best system that will meet their needs without breaking power/price budgets. They do not have the budget to deploy massive servers to each branch office, just to handle trivial server tasks (file storage, print serving, internal web document serving, etc). The systems they are considering need to be tested under load before a purchase determination can be made. Testing the software stack in a VM will not give them an accurate picture of how each system being considered will operate, and they want to be able to do 24/7 image deployment and load stress testing.

A new company is gearing up production on new servers for the market. These are low powered systems designed to provide low to medium server tasks on an extremely low power budget. The processor for these does not support virtualization, but the power consumption more than makes up for it, as a single 4U system with 20 4-core blades consumes less power than a single 8-core processor from a competing company that does support virtualization. They need to do extensive automated testing to help root out any hardware issues prior to production, part of which involves real-world environment testing. Having the ability to run automated tests already packaged for them would be a great benefit.

A software company, after months of testing software in an automated process based on virtualizing test systems. This has allowed them to speed up testing of their environment by a large factor with a lot of cost savings. However, when they finally release their product to the public, they are inundated with bug reports on real systems because of hardware/software configurations that their virtual environment couldn't account for. The hardware in question is relatively cheap and wouldn't have cost much to be able to add to their testing capabilities, averting a lot of these bug reports. The fixes to get the software running on these systems is also fairly quick, but public opinion has already spread like wildfire, diminishing their reputation.

Assumptions

Design

The environment should be modular so that as new types of systems and tests come online, they can be easily plugged in to the testing framework.

Implementation

One suggestion is to use Orchestra and Juju for image preparation and deployment over pxe on bare metal.

Test/Demo Plan

Take existing test plans (where possible) and break them into modular components that can be implemented on either kvm or bare metal test environments. Develop new test plans around a modular design.

Unresolved issues

Unchanged mindset of the existing developer community.

CategorySpec