Ubuntu Wiki

• Launchpad Entry: arm-m-image-building-console

• Created:

• Contributors: MichaelHudson, ScottBambrough

• Packages affected:

Summary

We will provide infrastructure to build images with a web based user interface. It will be possible for partners to install our software on their own premises, and eventually they will be able to use a Canonical-hosted instance of the software if they so desire.

Rationale

It is desirable to be able to create images on a reference system for sharing with others or releasing, aiding reproducibility and reducing unpredictability. For this reason, we will provide a system which can be set up for image building on request, allowing developers to trigger daily builds or milestone builds on the reference hardware.

User stories

Project lead Boris wants to build the final version of the operating system for his Internet tablet to send to the factory and build onto 100,000 units of hardware. He goes to the image building console, selects the final configuration and clicks go. A few minutes later, the image is available for download.

Assumptions

• Running code as root on our slave hardware is OK for now.

Design

• The UI of this tool will be a web application. As the image building tool we will be using only supports image building in a native environment, that is all the service can support for now.
• The application needs to support automatic daily builds as well as one-off manually triggered builds.
• The application will take as input for a build the URL of a branch that contains a configuration for the image building tool.

Implementation

Canonical have an internally-developed, currently proprietary, tool that does most of this, 'LexBuilder'. We will open source and base our work on this.

We will not be changing the basic architecture of this software. There are two major sections of the system, the master/slave part that builds the images and the UI part.

The master/slave part consists of a master daemon, only one of which is running for a given installation, and a number of slaves, which run on the hardware which is to build the images. These two daemons communicate via XML-RPC.

The UI is implemented as a web application implemented using Django. It allows users to request builds and set up automated daily builds.

The two parts communicate via a postgres database.

As well as producing the binary image, the tool will produce a manifest -- a record of what went into the image -- as part of the log.

The LexBuilder code has some issues that it need to be fixed before it can be turned into a deployable product:

• Currently image configuration is done by (effectively) specifying only the name of a branch on Launchpad -- the branch owner and project are hard coded.

  • Need to support arbitrary locations for live-helper configuration files for LexBuilder. The location of such configuration files is currently hard coded

  • Bug 582394 captures this requirement.
• Fix team access. Currently restricted to a particular Launchpad team.
  • First needs to be fixed to allow Launchpad team to be specified.
  • Then we need to look at a method of access control that doesn't depend on Launchpad.

• Fix deployment path. Currently LexBuilder is deployed to /srv.

  • We should deploy to /opt/linaro/lexbuilder for example.

  • This can be done currently but you have to pass a bunch of options to tell LexBuilder where things are.

    • Work needs to be done to make it easier to deploy to locations other then /srv
    • Ensure everything still works after the changes.
• Fix feature that lets users get the sources.list file for the project. Right now its a hard coded feature, that includes the username and password for the OEM internal development repository.
• clean up settings [ scottb: more detail required ]
• rewrite notifications [ scottb: more detail required ]
• Implement DB schema migration.

  • New features to LexBuilder will sometimes require changes to the database schema.

    • Implement use of South; a DB schema migration application for Django
    • Prevent users from having to manually patch their database when such new features are introduced.
• Dependencies are simply included wholesale into the project.
  • As part of the work to open source, these need to be removed
  • Code should use the distributions versions as dependencies
  • The packaging will need to take care of this.
• The model code is duplicated: one for the Django ORM and one for the Storm ORM.
  • The consensus decision from the sprint was to use the Django ORM.
• Setting up slaves is a little complicated. [ scottb: more detail required]

Migration

Would we want to migrate OEM services to using the version of the code we're developing?

This is a choice for OEM to make. Once LexBuilder is open sourced, it is hoped that new feature development will take place in the public release, and both Linaro and OEM can benefit from each others work.

Test/Demo Plan

None.

Unresolved issues

None.

TBD

If image building requires root then building in the build farm will require some kind of virtualization for ARM. Even if image building doesn't require root, this is probably a good idea in any case (unless we trust live-helper a lot). Do we block on this? (Essentially, Launchpad would need to support ARM ppas).

There is a spec about image building without root.

Actions

None.

CategorySpec