Ubuntu Wiki

• Launchpad Entry: rebuilder-framework

• Created: April 13th, 2009

• Contributors: Michael Casadevall

• Packages affected: dak(?), sbuild, buildd, wanna-build, gcc-wrapper

Summary

Being able to rebuild Ubuntu in a relatively easy fashion using different sets of CFLAGS and build environments is a useful tool but currently an extremely difficult task. A new build mechanism based off the existing free tools that is able to be setup, and then automatically track against a current release, building all packages as they are uploaded, or in the case of released version; tracking the updates component.

Release Note

TBD

Rationale

Having an automatic framework to handle rebuild when determining if an optimization should be adapted, or helping target specific hardware capabilities (such as ARMv5, ARMv6, and ARMv7 variant builds, or lpia -Os builds) would make having variant

Use Cases

• Alpha wants to determine if one set of CFLAGS vasty affects performance over another
• Beta wants to rebuild to enable address-space randomization and other experimental CFLAGS to create a hardened system.
• Gamma wishes to investigate the possibility of adding a new hardware subarchitecture (such as lpia with SSE extensions), and wishes to create a working prototype to catch potential issues ahead of time
• Delta wishes to port Ubuntu to a new architecture and has done a port of the base system.

Assumptions

• User is tracking Ubuntu and its structure of its archive
• User is knowledgeable w.r.t. to modifying packages and changing packages to create the desired rebuilt result.

Design

• We want to recycle as much code as possible. As Debian already has a freely available mass build system, we should base on that work
• Creation of metapackages, for the role of each machine
  • - Archive Master - machine holding archive - Build Master - machine hosted wanna-build/wanna-persue - Buildd - Actual build machine

Implementation

UI Changes

Most of the tools available use command line interfaces. wanna-purse is a fairly bland package build status and log interface that powers buildd.debian.org. It might be worth investing time to improve the UI tools.

Code Changes

Code changes broken down by section of build system:

dak

• Autoimport of overrides (essentially a crontab script)
• sync script to import new sources from target repo. Probably should also be able to handle binaries as well
• Trusted queue for allowing upload of unsigned changes and accepting it if there is no other reason to REJECT

wanna-build

• Take in account multiple archives when scanning for auto dep-waits
• Web-based UI intergrated into wanna-persue(?) (wishlist item at best)

buildd/sbuild

• Replace the maybe-successed/maybe-failed statuses with Successed/Failed. Failed builds should be auto-marked in wanna-build as such
• Requirement of email management of build system relaxed to a strong recommends when running a trusted buildd
• Support for trusted buildds which can upload unsigned changes

wanna-persue

• Overview pages of buildd status, and archive build status with a general failures page.
• Web-based retry/rescore interface(?)

How a package goes from being uploaded to being rebuilt by this framework

Items prefixed with a NI stands for needs-implementation and doesn't exist as of writing

• Package is uploaded to the Ubuntu archive and is published
• [NI as crontab] dak downloads and imports overrides from local mirrors
• [NI] dak scans a local mirror of the archive, downloads the source, and attempts imports it into target distros
  • - [NI] dak will import this into its trusted queue, and apply normal packaging check rules and either accept or reject
• dak publishes the package into the incoming queue
• wanna-build scans the incoming queue, and creates Needs-Build records for each distro/component if the package is not marked Not-For-Us
• wanna-build determines if build-deps can be met to the best of its ability, and marks the package Dep-Wait if they can't.
  • - [NI?] wanna-build takes into account build-dependencies available from Ubuntu but are not yet available in the rebuilt archive
• buildd polls wanna-build for packages to be built. Package status goes from Needs-Build to Building.
• buildd spawns sbuild instance against the appropriate chroot
• sbuild attempts to satisfy build-dependencies using the rebuilt archive first, and falling back on the main archive optionally
  • - This will be handled via APT pinning
• sbuild calls dpkg-buildpackage, and attempts to build the package, this process either successes or fails
• In either case, emails are sent from sbuild to wanna-purse host with build log, and buildd admin .
• [NI] In the case of failed builds, sbuild/buildd reports build status as Failed to wanna-build
• In case of failed build, buildd admin can respond by email: give-back, failed, dep-wait, not-for-us to change the status of the build record
• [NI] Changes file is automatically uploaded to rebuild archive if the buildd is trusted
• -OR- signed changes file is sent via email to the buildd
  • - buildd already has an auto-uploader, but it excepts a signed changes file by email, and will not attempt to upload without one
• Package is either uploaded to the unchecked queue (untrusted buildds), or trusted queue (trusted buildds).
• Package status goes from Building to Uploaded
• dak accepts or rejects the package
• Package enters into accepted queue, and is published at next dinstall
• Package status goes to Installed.
• Old versions are removed automatically via supersedation.

Migration

TBD

Test/Demo Plan

It's important that we are able to test new features, and demonstrate them to users. Use this section to describe a short plan that anybody can follow that demonstrates the feature is working. This can then be used during testing, and to show off after release. Please add an entry to http://testcases.qa.ubuntu.com/Coverage/NewFeatures for tracking test coverage.

This need not be added or completed until the specification is nearing beta.

Unresolved issues

Archive software

• Dak is the most featureful free Debian archive tool, but its extremely complex and difficult to setup as well as requiring a postgresql backend database
• Are any other simplier choices available that meets the requirements

Build system

• Do we *really* want to recycle the Debian build system (especially buildd)? A lot of it is old perl code, and somewhat hacky, but reimplementation would be a fairly massive chore

BoF agenda and discussion

Use this section to take notes during the BoF; if you keep it in the approved spec, use it for summarising what was discussed and note any options that were rejected.

CategorySpec