DailyUpstreamBuildsPOCSpec
|
Size: 4122
Comment:
|
Size: 7735
Comment: fix bullet formatting
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 5: | Line 5: |
| * '''Contributors''': JamesWestby, ColinWatson, MarkShuttleworth | * '''Contributors''': JamesWestby, MarkShuttleworth, MattZimmerman |
| Line 10: | Line 10: |
| Making packages available that represent the latest code available in the upstream version control system allows interested users and developers to follow the development much more closely. This allows them to improve the testing and feedback of the code that will one day become part of Ubuntu. | Packages that represent the latest code from a project lower the barrier to contribution to the project. They allow users and distribution developers to follow the latest code without having to build themselves from Bazaar. |
| Line 12: | Line 12: |
| As a step towards making this available for a lot of packages we will select a few packages to make available as daily builds in order to investigate the issues that arise from trying to automatically produce packages every day from changing code. | Vanilla packages of the latest code also provides an opportunity for confirming of bugs in conditions that will be most useful to upstream. This can ease the flow of bugs from distributions to upstream. Making it easy for people to set-up daily builds of the latest code will lead to more projects offering them. |
| Line 24: | Line 26: |
| * Brian is triaging a bug and believes it is an upstream problem, but the package the user is using is some way behind upstream's HEAD, and upstream like bugs to be tested against HEAD. Brian points the user to instructions for installing the daily builds of the package, and they test against a recent version and can then forward the bug if it is still an issue. | * Ryan runs an exciting new project, and wants to provide packages of the latest code to users in the easiest way possible. He doesn't want them to have to build from bzr every day, and would prefer to stay inside the packaging system so that dependencies can be easily satisfied for instance. He sets up a daily build of his code for Ubuntu and avertises it on the project page as the best way to get the code. * Brian is triaging a bug and believes it is an upstream problem, but the package the user is using is some way behind upstream's HEAD, and upstream like bugs to be tested against HEAD. Brian knows that there is a vanilla daily build of the package available, and so he asks the user to test with that. When the user confirms that it still happens the user files a helpful bug report upstream, and tests the fix the day after it lands. |
| Line 29: | Line 32: |
| * The packaging will only require changes in order to build/install fairly infrequently. If it requires intervention every day then automating the process is of little use. |
|
| Line 33: | Line 34: |
| You can have subsections that better describe specific parts of the issue. | The tool will work based on Bazaar branches, so that it can take advantage of bzrlib to do the heavy lifting. A "recipe" will specify the parts that go together to make up the source, including the addition of the packaging, and any patches that need to be applied. It will be a fairly simple format, specifying branches to merge (and optionally specific revisions of them). There will be a second permitted operation, "nesting" that will allow composite packages to be assembled from different branches. The tool will take a recipe file as input, and will follow the steps that it specifies, producing a source package, which will then be optionally signed and uploaded to a PPA. This then allows the command to be placed in cron to run every day, with modifcations done to the recipe, or to the branches it specifies as needed. As part of its operation the tool should generate a "manifest" and store it in the source packages. The manifest will show the exact revisions used from each branch, so that the package can be recreated later. There are a few other pieces of information that are needed at certain points in the operation of the tool, for instance the GPG key id to sign the package if that stage is needed. This information could live in the recipe, or in a separate info file. Often there will be no need to build the package if none of the branches have changed, but sometimes it may be desired to do this anyway, so that any problems caused by changes in the dependencies can be caught. There should be a way to specify which behaviour is wanted for each build. In order to provide the facility to not build the package if none of the branches have changed then the tool needs to have some way to know what revisions the branches were at in the previous build. Copies of the branches that were used in the last build would allow you to do this easily, however they could take up a significant amount of diskspace, and would constrain you to always building on the same machine. The alternative is to then keep around, or make available, the revisions that were used in the last build. |
| Line 37: | Line 53: |
| This section should describe a plan of action (the "how") to implement the changes discussed. Could include subsections like: | The recipe will be a simple text file, for example |
| Line 39: | Line 55: |
| === UI Changes === | {{{ branch lp:firefox merge lp:~fta/firefox/fix-build merge lp:~mozillateam/firefox/ubuntu.jaunty merge lp:~asac/firefox/fix-installation branch lp:xulrunner xulrunner merge lp:~asac/xulrunner/embed }}} |
| Line 41: | Line 64: |
| Should cover changes required to the UI, or specific UI that is required to implement this | where there are two operations, "branch" and "merge". It reads and works similar to a list of bzr commands, but there are some small differences. The main one is that "merge" is done in to the most recent "branch" target, not in to the root. |
| Line 43: | Line 66: |
| === Code Changes === | This is a more complex example. A basic recipe for most packages would just be |
| Line 45: | Line 68: |
| Code changes should include an overview of what needs to change, and in some cases even the specific details. | {{{ branch lp:project merge lp:ubuntu/jaunty/project }}} |
| Line 47: | Line 73: |
| === Migration === | with the more complex example showing how fixes can be applied where needed for the daily build. |
| Line 49: | Line 75: |
| Include: * data migration, if any * redirects from old URLs to new ones, if any * how users will be pointed to the new way of doing things, if necessary. |
The extra "branch" in the example shows how nesting can be used. In this case it embeds a copy of xulrunner in to the package, and then merges a branch that makes it suitable for embedding. This should be rarely used, but allows for more packages to be done in such a scheme. Specific revisions of the branches can be specified like so: {{{ branch lp:project merge lp:ubuntu/jaunty/project -r<revision> }}} where "revision" is any bzr revision specifier. The manifest can then simply be a recipe with revisions specified. For instance running the tool with the the second example may lead to the following manifest. {{{ branch lp:project -rrevid:foo@bar.com-12345 merge lp:ubuntu/jaunty/project -rrevid:fta@ubuntu.com-q2434fr }}} This means that the manifest can then be fed back in to the tool to reproduce the original steps. If given the the manifest of the previous run the tool can also detect whether there are changes in any of the branches. The manifest could either be stored somewhere known, or the tool could possibly extract it from the source package if it knows where to find it (e.g. in a PPA). |
| Line 56: | Line 99: |
| It's important that we are able to test new features, and demonstrate them to users. Use this section to describe a short plan that anybody can follow that demonstrates the feature is working. This can then be used during testing, and to show off after release. Please add an entry to http://testcases.qa.ubuntu.com/Coverage/NewFeatures for tracking test coverage. | In order to test the tool a few daily builds will be set-up and maintained. In addition to making sure the tool works well it will also show how much work is required to maintain daily builds, which is interesting information. |
| Line 58: | Line 101: |
| This need not be added or completed until the specification is nearing beta. | There is no need to have a contingency plan, as this work does not affect the distribution itself. The earliest the tool itself would end up in Ubuntu is in Juanty+1, and the daily-built packages won't go in to the repositories. |
| Line 62: | Line 105: |
| This should highlight any issues that should be addressed in further specifications, and not problems with the specification itself; since any specification with problems cannot be approved. == BoF agenda and discussion == Use this section to take notes during the BoF; if you keep it in the approved spec, use it for summarising what was discussed and note any options that were rejected. |
* Is there extra information needed about the recipe than just the steps, e.g. desired source package name, GPG key to sign with? Should this live in the recipe somewhere? * Is the described recipe format flexible enough? Is it possible to unambiguously talk about a location in the file to add extra steps? |
Launchpad Entry: daily-upstream-builds-poc
Created: 2009-02-03
Contributors: JamesWestby, MarkShuttleworth, MattZimmerman
Packages affected: Those chosen for the PoC, but no changes to the archive
Summary
Packages that represent the latest code from a project lower the barrier to contribution to the project. They allow users and distribution developers to follow the latest code without having to build themselves from Bazaar.
Vanilla packages of the latest code also provides an opportunity for confirming of bugs in conditions that will be most useful to upstream. This can ease the flow of bugs from distributions to upstream.
Making it easy for people to set-up daily builds of the latest code will lead to more projects offering them.
Release Note
This doesn't really have a place in the release notes as it doesn't change the distribution as such.
Rationale
Testing the HEAD of upstream's VCS can be very valuable, but building it is not easy for users, and is tedious to do regularly. Automatically producing these packages every day will alleviate some of this, but we would not have the resources to do this for every package currently. In order to investigate the complexities involved in doing it we will select a few packages and build a proof-of-concept that produces daily packages for them.
Use Cases
- Ryan runs an exciting new project, and wants to provide packages of the latest code to users in the easiest way possible. He doesn't want them to have to build from bzr every day, and would prefer to stay inside the packaging system so that dependencies can be easily satisfied for instance. He sets up a daily build of his code for Ubuntu and avertises it on the project page as the best way to get the code.
- Brian is triaging a bug and believes it is an upstream problem, but the package the user is using is some way behind upstream's HEAD, and upstream like bugs to be tested against HEAD. Brian knows that there is a vanilla daily build of the package available, and so he asks the user to test with that. When the user confirms that it still happens the user files a helpful bug report upstream, and tests the fix the day after it lands.
- Mary packages a fast moving project, and wants to keep on top of the changes upstream while Ubuntu is frozen for release. She installs the daily build packages in a test environment, and can test the changes and provide feedback to upstream so that the project is in good shape for the next Ubuntu release. She can also monitor for problems building the package to become aware of things like patches being merged upstream, and the required versions of dependencies changing.
Assumptions
Design
The tool will work based on Bazaar branches, so that it can take advantage of bzrlib to do the heavy lifting.
A "recipe" will specify the parts that go together to make up the source, including the addition of the packaging, and any patches that need to be applied. It will be a fairly simple format, specifying branches to merge (and optionally specific revisions of them). There will be a second permitted operation, "nesting" that will allow composite packages to be assembled from different branches.
The tool will take a recipe file as input, and will follow the steps that it specifies, producing a source package, which will then be optionally signed and uploaded to a PPA. This then allows the command to be placed in cron to run every day, with modifcations done to the recipe, or to the branches it specifies as needed.
As part of its operation the tool should generate a "manifest" and store it in the source packages. The manifest will show the exact revisions used from each branch, so that the package can be recreated later.
There are a few other pieces of information that are needed at certain points in the operation of the tool, for instance the GPG key id to sign the package if that stage is needed. This information could live in the recipe, or in a separate info file.
Often there will be no need to build the package if none of the branches have changed, but sometimes it may be desired to do this anyway, so that any problems caused by changes in the dependencies can be caught. There should be a way to specify which behaviour is wanted for each build.
In order to provide the facility to not build the package if none of the branches have changed then the tool needs to have some way to know what revisions the branches were at in the previous build. Copies of the branches that were used in the last build would allow you to do this easily, however they could take up a significant amount of diskspace, and would constrain you to always building on the same machine. The alternative is to then keep around, or make available, the revisions that were used in the last build.
Implementation
The recipe will be a simple text file, for example
branch lp:firefox merge lp:~fta/firefox/fix-build merge lp:~mozillateam/firefox/ubuntu.jaunty merge lp:~asac/firefox/fix-installation branch lp:xulrunner xulrunner merge lp:~asac/xulrunner/embed
where there are two operations, "branch" and "merge". It reads and works similar to a list of bzr commands, but there are some small differences. The main one is that "merge" is done in to the most recent "branch" target, not in to the root.
This is a more complex example. A basic recipe for most packages would just be
branch lp:project merge lp:ubuntu/jaunty/project
with the more complex example showing how fixes can be applied where needed for the daily build.
The extra "branch" in the example shows how nesting can be used. In this case it embeds a copy of xulrunner in to the package, and then merges a branch that makes it suitable for embedding. This should be rarely used, but allows for more packages to be done in such a scheme.
Specific revisions of the branches can be specified like so:
branch lp:project merge lp:ubuntu/jaunty/project -r<revision>
where "revision" is any bzr revision specifier.
The manifest can then simply be a recipe with revisions specified. For instance running the tool with the the second example may lead to the following manifest.
branch lp:project -rrevid:foo@bar.com-12345 merge lp:ubuntu/jaunty/project -rrevid:fta@ubuntu.com-q2434fr
This means that the manifest can then be fed back in to the tool to reproduce the original steps.
If given the the manifest of the previous run the tool can also detect whether there are changes in any of the branches. The manifest could either be stored somewhere known, or the tool could possibly extract it from the source package if it knows where to find it (e.g. in a PPA).
Test/Demo Plan
In order to test the tool a few daily builds will be set-up and maintained. In addition to making sure the tool works well it will also show how much work is required to maintain daily builds, which is interesting information.
There is no need to have a contingency plan, as this work does not affect the distribution itself. The earliest the tool itself would end up in Ubuntu is in Juanty+1, and the daily-built packages won't go in to the repositories.
Unresolved issues
- Is there extra information needed about the recipe than just the steps, e.g. desired source package name, GPG key to sign with? Should this live in the recipe somewhere?
- Is the described recipe format flexible enough? Is it possible to unambiguously talk about a location in the file to add extra steps?
DailyUpstreamBuildsPOCSpec (last edited 2009-10-26 13:14:21 by eth0)