Ubuntu Wiki

• Launchpad Entry: server-lucid-automated-testing

• Created: 2009-11-23

• Contributors:

• Packages affected:

Summary

This should provide an overview of the issue/functionality/change proposed here. Focus here on what will actually be DONE, summarising that so that other people don't have to read the whole spec. See also CategorySpec for examples.

Release Note

A considerable amount of time has been spent on setting up automating testing of the server product. It is our hope that this will provide a more solid release of Ubuntu Server than ever before.

Rationale

Testing is a tedious, thankless job. Thankfully, a lot of things can be done automatically. Many packages have test suites that we're not running (for one reason or another), we have qa-regression-tests, and lots of other means for testing things without minimal day to day effort.

User stories

1. Soren uploads a new version of dovecot, but is worried he might break compatibility with some IMAP client. However, he sleeps well that evening, knowing that the automatic regression testing suite will raise an alert over night if something broke. 1. Jamie notices a bug in libvirt and works on a patch. While test-building the package locally, the test suite informs him that he broke a little-used feature in an obscure corner of the lxc driver in libvirt. He promptly fixes it, the test suite is happy again, and Jamie can proceed to upload the package to Ubuntu.

Assumptions

Design

The goal is to detect as many problems as early as possible.

• Many packages ship test suites.
  • Some of these can be run at build time. We should make them do so.
  • Other packages ship test suites that can't easily be run at build time. We should arrange for them to be run daily "somewhere" and somehow get alerted about failures (regressions).
• The security team and QA team have a series of tests they use to ensure they don't introduce regressions in stable releases.
  • We should use these during development as well. This should be done daily and we should get a report back about failures.
• We want to be alerted about performance regressions as well.

Implementation

qa-regression-testing scripts

We will integrate the security team's qa-regression-test collection into checkbox, and have it run on a daily basis. Feedback will be collected by the QA team and turned into bugs for the server team to deal with.

Performance testing

The Phoronix test suite seems to be reasonably comprehensive. We will run it on a daily basis and keep an eye on performance regressions. Of course this needs to run on the same hardware every time.

Upstream test suites

A number of server packages are known to provide test suites:

• postgresql

[to be continued]

Implementation

This section should describe a plan of action (the "how") to implement the changes discussed. Could include subsections like:

UI Changes

Should cover changes required to the UI, or specific UI that is required to implement this

Code Changes

Code changes should include an overview of what needs to change, and in some cases even the specific details.

Migration

Include:

• data migration, if any
• redirects from old URLs to new ones, if any
• how users will be pointed to the new way of doing things, if necessary.

Test/Demo Plan

It's important that we are able to test new features, and demonstrate them to users. Use this section to describe a short plan that anybody can follow that demonstrates the feature is working. This can then be used during testing, and to show off after release. Please add an entry to http://testcases.qa.ubuntu.com/Coverage/NewFeatures for tracking test coverage.

This need not be added or completed until the specification is nearing beta.

Unresolved issues

This should highlight any issues that should be addressed in further specifications, and not problems with the specification itself; since any specification with problems cannot be approved.

BoF agenda and discussion

Automated testing is a great way to prevent/detect regressions.

Security team qa-regression-testing scripts:

• currently integrating in checkbox - aim at 80% - the most easy ones.
• cr3 could run the tests in the data centre

What we want:

• every day a report is generated covering which tests have been run and their results

Running tests in EC2.

Test results reporting:

• leverage checkbox.
• checkbox supports different submission plugin. What to use to track the results and generate reports?

Inclusion in milestone reports presented during the release meeting team.

QA team: easy to run the tests and process the test results internally (black box).

• tests are run and failures are reported as bug by the QA team.

How are test suites updated because of changes in the system? Who?

• QA team finds out about the failure and reports the bug
• QA team fixes the test and writes tests.

What needs testing?

Integration list:

1. qa-regression-testing scripts
   1. enable selected phoronix tests
2. upstream test suites
   • integrate postgresql test suite
   • integrate puppet-testsuite suite package
   • integrate dovecot imap test suite (not packaged)
   • apache tests (has a framework, use documented in QRT)
   • libvirt test suite (not run during build, but could be), also tests in QRT (but not python-unit)
   • mysql test suite runs during the build
   • openldap test suite runs during the build
   • cups test suite runs in the build, but has concept of other levels (eg smbtorture)
   • samba
     • 'make test', but needs to be built with --enable-socket-wrapper
     • smbtorture
   • php5
3. integrate iso testing tests in checkbox:

   • ServerWhole

4. review all the packages on the server CD
5. Multi-system environements: documentation.
   • pacemaker
   • drbd

What sort of testing do we want to perform?

• Stress/performance testing?
  • E.g. check if Apache suddenly can handle much fewer requests per second
    • than the previous day?
  • leverage phoronix test suite?
• Functional testing?
  • E.g. use different mail clients to talk to a mail server?
  • Try a suite of different configuration combinations that we know used to
    • work?
• Upgrade testing?
  • Do a very fat hardy install (all sorts of different servers, clients, and
    • other stuff), and upgrade it to Lucid and see how it breaks?
  • Repeat for different configurations? mvo testing infrastructure: only looking at package upgrade failure. How to test that services are working correctly after the upgrade? Marjo to figure it out.
• Enabling test suite if they have one

CategorySpec