ARMImageBuildingConsole

Revision 5 as of 2010-05-28 18:03:08

Clear message

Summary

We will provide infrastructure to build images with a web UI.

It will be possible for partners to install our software on their own hardware, and eventually use a Canonical-hosted instance of the software (once there is a virtualized/emulated ARM build farm).

Rationale

It is desirable to be able to create images on a reference system for sharing with others or releasing, aiding reproducibility and reducing unpredictability. For this reason, we will provide a system which can be set up for image building on request, allowing developers to trigger daily builds or milestone builds on the reference hardware.

User stories

Project lead Boris wants to build the final version of the operating system for his internet tablet to send to the factory and build onto 100,000 units of hardware. He goes to the image building console, selects the final configuration and clicks go. A few minutes later, the image is available for download.

Assumptions

  • Running code as root on our slave hardware is OK for now.

Design

The UI of this tool will be a web application.

As the image building tool we will be using only supports image building in a native environment, that is all the service can support for now.

The application needs to support automatic daily builds as well as one-off manually triggered builds.

The application will take as input for a build the URL of a branch that contains a configuration for the image building tool.

XXX This section should probably be expanded to explain the other requirements. Unfortunately I don't know what they are.

Implementation

Canonical have an internally-developed, currently proprietary, tool that does most of this, 'lexbuilder'. We will open source and base our work on this.

We will not be changing the basic architecture of this software. There are two major sections of the system, the master/slave part that builds the images and the UI part.

The master/slave part consists of a master daemon, only one of which is running for a given installation, and a number of slaves, which run on the hardware which is to build the images. These two daemons communicate via XML-RPC.

The UI is implemented as a web application implemented using Django. It allows users to request builds and set up automated daily builds.

The two parts communicate via a postgres database.

As well as producing the binary image, the tool will produce a manifest -- a record of what went into the image -- as part of the log.

The lexbuilder code has some issues that it need to be fixed before it can be turned into a deployable product:

  • Dependencies are simply included wholesale into the project
  • The model code is duplicated: one for the Django ORM and one for the Storm ORM.
    • Probably just using the Django ORM makes sense.
  • Currently image configuration is done by (effectively) specifying only the name of a branch on Launchpad -- the branch owner and project are hard coded.
    • We should probably allow the application to define a project, and a project will specify the full URL of a branch containing the config.
  • Setting up slaves is a little complicated.

We will want to package our software into master, slave and webapp debs for ease of installation.

Migration

Would we want to migrate OEM services to using the version of the code we're developing?

Test/Demo Plan

-

Unresolved issues

TBD

If image building requires root then building in the build farm will require some kind of virtualization for ARM. Even if image building doesn't require root, this is probably a good idea in any case (unless we trust live-helper a lot). Do we block on this? (Essentially, Launchpad would need to support ARM ppas).

There is a spec about image building without root.

Actions

This is from the gobby notes, but they are (a) a bit vague and (b) not assigned to anyone, so I don't know how to weave them into the spec.

  • virtual ppa infrastructure needed for ARM because image building requires building parts as root
    • virtualized QEMU is possible but slow.

CategorySpec