Ubuntu Wiki

• Launchpad Entry: daily-upstream-builds-poc

• Created: 2009-02-03

• Contributors: JamesWestby, MarkShuttleworth, MattZimmerman

• Packages affected: Those chosen for the PoC, but no changes to the archive

Summary

Packages that represent the latest code from a project lower the barrier to contribution to the project. They allow users and distribution developers to follow the latest code without having to build themselves from Bazaar.

Vanilla packages of the latest code also provides an opportunity for confirming of bugs in conditions that will be most useful to upstream. This can ease the flow of bugs from distributions to upstream.

Making it easy for people to set-up daily builds of the latest code will lead to more projects offering them.

Release Note

This doesn't really have a place in the release notes as it doesn't change the distribution as such.

Rationale

Testing the HEAD of upstream's VCS can be very valuable, but building it is not easy for users, and is tedious to do regularly. Automatically producing these packages every day will alleviate some of this, but we would not have the resources to do this for every package currently. In order to investigate the complexities involved in doing it we will select a few packages and build a proof-of-concept that produces daily packages for them.

Use Cases

• Ryan runs an exciting new project, and wants to provide packages of the latest code to users in the easiest way possible. He doesn't want them to have to build from bzr every day, and would prefer to stay inside the packaging system so that dependencies can be easily satisfied for instance. He sets up a daily build of his code for Ubuntu and avertises it on the project page as the best way to get the code.
• Brian is triaging a bug and believes it is an upstream problem, but the package the user is using is some way behind upstream's HEAD, and upstream like bugs to be tested against HEAD. Brian knows that there is a vanilla daily build of the package available, and so he asks the user to test with that. When the user confirms that it still happens the user files a helpful bug report upstream, and tests the fix the day after it lands.
• Mary packages a fast moving project, and wants to keep on top of the changes upstream while Ubuntu is frozen for release. She installs the daily build packages in a test environment, and can test the changes and provide feedback to upstream so that the project is in good shape for the next Ubuntu release. She can also monitor for problems building the package to become aware of things like patches being merged upstream, and the required versions of dependencies changing.

Assumptions

Design

The tool will work based on Bazaar branches, so that it can take advantage of bzrlib to do the heavy lifting.

A "recipe" will specify the parts that go together to make up the source, including the addition of the packaging, and any patches that need to be applied. It will be a fairly simple format, specifying branches to merge (and optionally specific revisions of them). There will be a second permitted operation, "nesting" that will allow composite packages to be assembled from different branches.

The tool will take a recipe file as input, and will follow the steps that it specifies, producing a source package, which will then be optionally signed and uploaded to a PPA. This then allows the command to be placed in cron to run every day, with modifcations done to the recipe, or to the branches it specifies as needed.

As part of its operation the tool should generate a "manifest" and store it in the source packages. The manifest will show the exact revisions used from each branch, so that the package can be recreated later.

There are a few other pieces of information that are needed at certain points in the operation of the tool, for instance the GPG key id to sign the package if that stage is needed. This information could live in the recipe, or in a separate info file.

Often there will be no need to build the package if none of the branches have changed, but sometimes it may be desired to do this anyway, so that any problems caused by changes in the dependencies can be caught. There should be a way to specify which behaviour is wanted for each build.

In order to provide the facility to not build the package if none of the branches have changed then the tool needs to have some way to know what revisions the branches were at in the previous build. Copies of the branches that were used in the last build would allow you to do this easily, however they could take up a significant amount of diskspace, and would constrain you to always building on the same machine. The alternative is to then keep around, or make available, the revisions that were used in the last build.

Implementation

The recipe will be a simple text file, for example

branch lp:firefox
merge lp:~fta/firefox/fix-build
merge lp:~mozillateam/firefox/ubuntu.jaunty
merge lp:~asac/firefox/fix-installation
branch lp:xulrunner xulrunner
merge lp:~asac/xulrunner/embed

where there are two operations, "branch" and "merge". It reads and works similar to a list of bzr commands, but there are some small differences. The main one is that "merge" is done in to the most recent "branch" target, not in to the root.

This is a more complex example. A basic recipe for most packages would just be

branch lp:project
merge lp:ubuntu/jaunty/project

with the more complex example showing how fixes can be applied where needed for the daily build.

The extra "branch" in the example shows how nesting can be used. In this case it embeds a copy of xulrunner in to the package, and then merges a branch that makes it suitable for embedding. This should be rarely used, but allows for more packages to be done in such a scheme.

Specific revisions of the branches can be specified like so:

branch lp:project
merge lp:ubuntu/jaunty/project -r<revision>

where "revision" is any bzr revision specifier.

The manifest can then simply be a recipe with revisions specified. For instance running the tool with the the second example may lead to the following manifest.

branch lp:project -rrevid:foo@bar.com-12345
merge lp:ubuntu/jaunty/project -rrevid:fta@ubuntu.com-q2434fr

This means that the manifest can then be fed back in to the tool to reproduce the original steps.

If given the the manifest of the previous run the tool can also detect whether there are changes in any of the branches. The manifest could either be stored somewhere known, or the tool could possibly extract it from the source package if it knows where to find it (e.g. in a PPA).

Test/Demo Plan

In order to test the tool a few daily builds will be set-up and maintained. In addition to making sure the tool works well it will also show how much work is required to maintain daily builds, which is interesting information.

There is no need to have a contingency plan, as this work does not affect the distribution itself. The earliest the tool itself would end up in Ubuntu is in Juanty+1, and the daily-built packages won't go in to the repositories.

Unresolved issues

• Is there extra information needed about the recipe than just the steps, e.g. desired source package name, GPG key to sign with? Should this live in the recipe somewhere?
• Is the described recipe format flexible enough? Is it possible to unambiguously talk about a location in the file to add extra steps?

CategorySpec