Summary

Members of the foundations team are creating infrastructure to test critical software in the distribution. This work should be continued to expand coverage. Steps should also be taken to share overlapping infrastructure and centralize publishing of data where possible.

Release Note

Rationale

User stories

Assumptions

We will be able to run Hudson on Canonical-owned, Internet-facing hardware.

Design

A public facing instance of Hudson will collect test results from a set of slave machines, both physical and virtual, or possibly from a private Hudson that speaks to datacenter-located physical hardware. This will at a minimum run tests for the installer and provide detailed data on their results.

For other projects we will investigate migrating them to this system, or if the project's tests do not fit the Hudson model, we may use Hudson to consume the results of these tests and provide a common interface for tracking their output.

Implementation

Centralized testing

Hudson provides an API for exposing its results to external sources. We may use this to connect automatic installer tests and automatic upgrade tests to the ISO testing tracker, but will do so only if deemed necessary.

Installer testing

A Hudson instance will be configured with a set of jobs that will do the following:

For security, we could run two instances of Hudson, one that lives behind the firewall and talks to the private slaves, and another that's public facing and receives results from the private Hudson via the Build Publisher Plugin

Hudson's concepts of axes can be leveraged to easily integrate further types of live environment tests, like preseeded tests.

Package upload testing

A PPA will created in Launchpad with permissions that match that of the Ubuntu archive. A bot will monitor this PPA and run tests on any packages in it, then copy the packages into the archive if the tests pass.

piuparts may be used at as one possible source of tests, provided it does not generate too many false positives.

Depending on how useful the initial implementation turns out to be, the documentation and default dput configuration could be changed to point at this PPA. A clear note that this test system can be subverted by uploading straight to the Ubuntu archive.

Upgrade testing

Mago Tests

After a successful installation, it would be nice to run GUI tests to show that the installed system is complete.

As Mago reports in both HTML and XML, it is possible to attach the report to the Hudson results using the HTML plugin. They HTML plugin can be easily enabled through "Manage Hudson" -> "Manage Plugins".

Xvfb is set up to be able to run the GUI tests:

$ /usr/bin/Xvfb :5 -ac -screen 0 1024x768x8

As a proof of concept, this the the config.xml file needed to run the gedit tests:

<?xml version='1.0' encoding='UTF-8'?>
<project>
  <actions/>
  <description></description>
  <keepDependencies>false</keepDependencies>
  <properties/>
  <scm class="hudson.scm.NullSCM"/>
  <canRoam>true</canRoam>
  <disabled>false</disabled>

  <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
  <triggers class="vector"/>
  <concurrentBuild>false</concurrentBuild>
  <builders>
    <hudson.tasks.Shell>
      <command>rm -rf mago
bzr branch lp:mago
cd mago
DISPLAY=:5.0 PYTHONPATH=. ./bin/mago --noa11y --target=/var/lib/hudson/userContent/mago -a gedit</command>
    </hudson.tasks.Shell>

  </builders>
  <publishers>
    <htmlpublisher.HtmlPublisher>
      <reportTargets>
        <htmlpublisher.HtmlPublisherTarget>
          <reportName>Mago Report</reportName>
          <reportDir>/var/lib/hudson/userContent/mago/gedit</reportDir>
          <reportFiles>gedit_chains.html</reportFiles>

          <keepAll>true</keepAll>
          <wrapperName>htmlpublisher-wrapper.html</wrapperName>
        </htmlpublisher.HtmlPublisherTarget>
      </reportTargets>
    </htmlpublisher.HtmlPublisher>
  </publishers>
  <buildWrappers/>
</project>

Unresolved issues


CategorySpec

FoundationsTeam/Specs/SharingTestingInfrastructure (last edited 2010-06-10 07:28:35 by apulido)