ARMImageBuildingTool

Summary

We will provide and document an easy-to-use tool to build system images.

This is as much a matter of providing examples, writing documentation, establishing conventions and best practice and social engineering as it is software development task. There are variety of tools for this already, but we will pick one and polish it until it is suitable to become a standard.

Rationale

Being able to easily build images is required to make developing on ARM easy, as it it sometimes the only way to test a fix. In addition, because ARM has so many flavors that require different images it means we need something scalable and extensible. The existing landscape of tool-building images consists of a wide variety of scripts of various levels of generality and quality. The aim is to pick a chosen tool to put in the hands of developers, and then improve it such that images can be created with minimum developer effort.

User stories

Developer John has made some changes to the kernel and needs to build an image to flash onto a SD card so he can test boot times on his prototype hardware. He makes a simple change to a configuration file, runs a shell command and use dd to blat the image onto the SD card.

Assumptions

  • We will have the ability to run code as root during the image build process (although see this spec)

  • We assume the image will be built in its native environment, either on ARM hardware or using emulation.

Design

We will standardise on live-helper.

The issues identified that are most pressing for us to improve in the tool are support for ARM bootloaders, improved configuration, and filtering files at package install time.

As well as producing the binary image, the tool will produce:

  • a manifest -- a record of what went into the image
  • a license report -- a record of the licenses used by each item in the manifest
  • a build log

Bootloaders

live-helper does not yet support ARM boot-loaders, so we need to extend it to be able to create images using these bootloaders. It already has support for multiple bootloaders, so it will be a case of adding some more options. There will be some scope for automatic detection (is the architecture "armel", but user selection should be allowed too). We aim to standardise on u-boot, so that will be the bootloader that we default to on armel.

It also needs to support creating images with multiple partitions, which are required on some ARM targets due to constraints around booting.

Configuration

The current live-helper configuration format is rather verbose, with lots of files and directories being created to just have the defaults. This makes it harder for developers to see what is specific to their project, and what is just defaults.

Therefore we would like to be able to use a simplified configuration format that can just be a minimal configuration for each project if desired.

While this makes it easier to comprehend, it does lose the self-documenting of other options that are currently defaults. Therefore we will also ensure that these are documented elsewhere, not just in the code. At first this may just be providing access to a full default configuration for comparison, but there may be better ways to present the information.

The first step will be to ensure that live-helper can work from a minimised configuration file.

After that the creation of a new project should be changed to output a minimal file, rather than include all the defaults.

Once we have that we can discuss with developers whether they want an alternative format. Currently the format is very low-level, requiring you to look at the details. It's possible that a higher-level configuration interface would be desired, breaking images in to classes or similar, such that you can pick a class and get several options set correctly for that class, but still with the ability to tweak specific options.

Filtering Files

In order to get a smaller footprint for some applications the developer may want to filter the files installed from the packages, to remove /usr/share/doc for instance.

The developer will be able to select a list of paths to filter as packages are installed, assuming that this functionality is available in dpkg.

Implementation

Bootloaders

There needs to be a small live-helper script created, lh_binary_uboot, along the same lines as the other bootloader scripts (lh_binary_grub etc.) This will install uboot as the bootloader for the image.

Then live-helper needs to be modified to default to using the script on armel if the choice of bootloader isn't overridden.

live-helper needs some refactoring and code to allow to allow it to build images with multiple partitions. It will also need to be able to read multiple-partition requirements from the configuration.

Configuration

First we check that live-helper can work from a minimal configuration. This involves testing and checking the code to see if it assumes that variables are set, rather than checking whether they are empty before using them. Unfortunately this is not easy to do programatically, but there isn't too much to inspect by eye.

The the code that outputs skeleton configurations should then be changed to output a minimal one, possibly with an option to retain the old behaviour.

If it is decided that a higher-level configuration interface is wanted then that will have to be designed based on the experience of developers.

Filtering Files

This part of the spec is dependent on having filtering support in dpkg.

Once that is available we can add a configuration option to live-helper that will specify the exclude/include patterns that dpkg will use. live-helper will then be changed to put these options in the appropriate file when building the image, in turn causing dpkg to filter those files out when installing.

Migration

We're only targeting the use of the new tool at new projects.

Test/Demo Plan

Bootloaders

There will be tests done to ensure that uboot can be installed by default, by explicit selection, and can be overriden to install something else on armel.

Also where there are options for the way that it is installed each will be tested to ensure that the choice is passed through from the configuration file to the right options on disk.

Configuration

There will be testing involved in the first stage to ensure that a minimal configuration can be used, and then testing of the output of the project creation step once the second stage has been completed.

Filtering Files

We will test that various combinations of include and exclude have the desired effect on the image.

Unresolved Issues

UDS discussion

There seemed to be consensus that the tool would be built on live-helper, which despite being awkward to configure and not yet supporting ARM bootloaders, is maintained and flexible, has decent docs and a sane architecture, and generally seemed the most promising.

Other bases considered were:

  • rootstock (only builds filesystems, not images)
  • livecd-rootfs (requires two phases, basically unconfigurable)
  • vmbuilder (not so good at images requiring specific partition layout)

All tools are built on debootstrap. It might make sense to base them on multistrap instead:

  • - it uses apt+dpkg instead of reimplementing parts - can use multiple archives - qemu-debootstrap exists, and should perhaps be ported. - live-helper has plans to use multistrap. - it does mean you have problems with pre-depends and dpkg-diverts - multiple archives is nice, but there are other ways to do that, and
    • live-helper makes that easy.

live-helper v2 is in development and although only alpha probably makes sense as the base of our system.

live-helper configuration is too complex to get started. Abstractions might be good.

  • - perhaps think dh v7, where the tool does the sensible thing by default, and you can just add overrides where you have unusual needs. - Many files/folders by default, providing a way to have a single configuration file that only specifies the necessary things would make it easier to grasp what the project requires.
    • - it probably makes sense to try to get these changes done upstream

Reproducing previous builds given the same inputs is desirable but tricky.

  • - OEM finds that some companies want to be able to do it. - Want to be able to make a single change. - Snap-shots (like snapshot.debian.org) could be an option but requires disk space.

Actions

This is from the gobby notes, but they are (a) a bit vague and (b) not assigned to anyone, so I don't know how to weave them into the spec.

  • virtual ppa infrastructure needed for ARM because image building requires building parts as root
    • virtualized QEMU is possible but slow.
  • add support for ARM bootloaders to live-helper
  • write a proposal for image building service (DB, web ui, etc.)
  • Look into livehelper 2
  • Investigate easier config for live-helper
  • Simplification of live-helper usage (configuration) for common targets.

Discussion

asac: uboot feature is already in my live-helper branch in a lh_binary-bootparts script. it automatically generated uimage etc. pieces if the linux flavour is supported - lp:~asac/live-helper/trunk.virtual-hdd - taking that work item to maybe sort this out in a separate script.

asac: i think the minimal configuration should be a meta config file format that then maps to the lh config invocation to create the config tree on the fly; I have such a script in the chessy configs for minimal and netbook; we should just think about what options we want to make available and then make a meta config format that creates the proper config command. From my findings having the config tree in bzr is kind of overkill; at best we have just the meta config file and maybe offer a few hook directories for chroot_local-packages etc.

mwhudson: I think this is essentially the question of do we extend lh to change the way it is configured or wrap it, so that we generate a config today's lh can understand from something else? I've always assumed we'd be doing the former, but possibly for no very good reason.

cody-somerville: OEM Services loves having project config in bzr branches as we create a 'base' config branch that we branch off of. It makes making the same change to all our projects easier thanks to bzr merge. The revision control of the config is also been very beneficial on a number of occasions. However, look at the 'auto' scripts feature in live-helper as a possible way to implement the simplified config.

asac: also i am partially negative on the idea of teaching live-helper to do multi partition images; could be that i am wrong, but from my experience its really fragile to do that because you dont know about cylinder/cluster/sector metrics if you do that on a virtual .img. i would suggest that we rather focus on creating rootfs+partition and a to-media script like the setup_sdcard.sh we made recently. lets discuss this.

cody-somerville: Alex and I discussed this and we're going to implement both. Basically, the binary helpers will create each independent chunk. Then when we get to the image helpers they'll just do the right thing depending on if we're building a tarball, usb-hdd, or something else (ie. tarball each chunk, thread each chunk together into an image with each chunk having its own partition, or something else depending on the requested output). We'll also make the assumption that images are going to be used as installer whereas tarballs in some situations might just be extracted onto system.

daniel: asac, my plan is to teach lh to understand partman recipies at some point, that way the same configuration mechanism that can be used to preseed d-i can be used for lh too.; also, if possible, i'd really like to avoid people doing 'meta' things or 'wrappers' arround lh, and just solve whatever they need in order to use lh directly. if you have a good idea on what/how to change in the config structure, i'm happy to help make it possible. it would be nice if you could have a look about the auto/config things, it might be, as cody already said, already be a solution for your problem. if not, it would be nice if you could explain me what your actual problem is/use cases are that you need to solve, since i couldn't really understand it form the things i was pointed to. about the CHS problem, is this really a problem these days, don't they have all the same specs anyway?

CategorySpec

Specs/M/ARMImageBuildingTool (last edited 2010-07-09 18:43:55 by 204)