ARMImageBuildingConsole

Revision 1 as of 2010-05-26 03:55:07

Clear message

Summary

There are two aspects to this spec:

  • providing a tool to build system images
  • infrastructure to build images with a web UI

It almost certainly makes sense to separate these into two specs (and maybe only do the first one for Maverick).

Release Note

?

Rationale

Being able to easily build images is required to make developing on ARM easy, as it it sometimes the only way to test a fix. In addition, because ARM has so many flavours that require different images it means we need something scalable and extensible. The existing landscape of tool-building images consists of a wide variety of scripts of various levels of generality and quality. The aim is to pick a chosen tool to put in the hands of developers, and then improve it such that images can be created with minimum developer effort.

In addition, it is desirable to be able to create images on a reference system for sharing with others or releasing, aiding reproducibility and reducing unpredictability. For this reason, we will provide a system which can be set up for image building on request, allowing developers to trigger daily builds or milestone builds on the reference hardware.

User stories

Developer John has made some changes to the kernel and needs to build an image to flash onto a SD card so he can test boot times on his prototype hardware. To do this, he runs debuild(-cross??) to build the kernel using a cross-compiler, edits his image building configuration to reference the newly built kernel, runs lh build or similar (which uses qemu under the hood) and uses dd to blat the image onto the sd card.

Project lead Boris wants to build the final version of the operating system for his internet tablet to send to the factory and build onto 100,000 units of hardware. He goes to the image building console in Launchpad, selects the final configuration and clicks go. A few minutes later, the image is available for download.

Assumptions

  • We will have the ability to run code as root during the image build process.
  • We assume the image will be built in its native environment, either on ARM hardware or using emulation.

Design

Tool

There seemed to be consensus that the tool would be built on live-helper, which despite being awkward to configure and not yet supporting ARM bootloaders, is maintained and flexible, has decent docs and a sane architecture, and generally seemed the most promising.

Other bases considered were:

  • rootstock (only builds filesystems, not images)
  • livecd-rootfs (requires two phases, basically unconfigurable)
  • vmbuilder (not so good at images requiring specific partition layout)

All tools are built on debootstrap. It might make sense to base them on multistrap instead:

  • - it uses apt+dpkg instead of reimplementing parts - can use multiple archives - qemu-debootstrap exists, and should perhaps be ported. - live-helper has plans to use multistrap. - it does mean you have problems with pre-depends and dpkg-diverts - multiple archives is nice, but there are other ways to do that, and
    • live-helper makes that easy.

live-helper v2 is in development and although only alpha probably makes sense as the base of our system.

live-helper configuration is too complex to get started. Abstractions might be good.

  • - perhaps think dh v7, where the tool does the sensible thing by default, and you can just add overrides where you have unusual needs. - Many files/folders by default, providing a way to have a single configuration file that only specifies the necessary things would make it easier to grasp what the project requires.
    • - it probably makes sense to try to get these changes done upstream

Reproducing previous builds given the same inputs is desirable but tricky.

  • - OEM finds that some companies want to be able to do it. - Want to be able to make a single change. - Snap-shots (like snapshot.debian.org) could be an option but requires disk space.

Infrastructure

It's probably possible to use qemu to build images on intel hardware should we want to (will be very slow though, obviously).

Building images will chew up a lot of disk space very quickly.

There is a need to store a manifest -- a record of what went into the image.

Web UI for creating builds and seeing the logs etc.

Be able to go from image to the configuration that created it.

Implementation

Tool

Based on live-helper.

Infrastructure

Based on Launchpad?

OEM services have done lots of Canonical-internal development here, at the very least we should talk to them about that, if we can't reuse it for some reason.

Migration

Would we want to migrate OEM services to using our new infrastructure?

Test/Demo Plan

-

Unresolved issues

TBD

If image building requires root then building in the build farm will require some kind of virtualization for ARM. Even if image building doesn't require root, this is probably a good idea in any case (unless we trust live-helper a lot). Do we block on this? (Essentially, Launchpad would need to support ARM ppas).

There is a spec about image building without root, although the gobby notes for that spec seem to talk about rootstock, not live-helper. Presumably, changes to not require root would require hacking live-helper a bit.

Actions

This is from the gobby notes, but they are (a) a bit vague and (b) not assigned to anyone, so I don't know how to weave them into the spec.

  • virtual ppa infrastructure needed for ARM because image building requires building parts as root
    • virtualized QEMU is possible but slow.
  • add support for ARM bootloaders to live-helper
  • write a proposal for image building service (DB, web ui, etc.)
  • Look into livehelper 2
  • Investigate easier config for live-helper
  • Simplification of live-helper usage (configuration) for common targets.

CategorySpec