public-panda-ppa-build-cluster

Summary

As the ARM community grows we need secure Public PPA build machines. This is harder to achieve than it initially sounds as there is no cost effective virtualization for ARM CPU's at the current time. This spec outlines a method of mechanical virtualization that allows the same functionality as is achieved on the X86 PPA's but doing so without virtualization.

Release Note

The impact of this is that the data center (and anyone that wants to build the hardware) can deploy a cluster of ARMv7 Cortex A9 dual core build machines in a 4U rack mounted case. This will allow Ubuntu to offer ARM PPA's and also to allow the ARM arch to have more and faster builders and add more as necessary.

Rationale

User stories

  • Joe wants to provide his generic software package on ARM hardware supported by Ubuntu as well as X86 hardware, the PPA needs to support selecting ARM as a supported architecture for package builds.

    Jane wants to test her ARM driver and needs to build the package for testing by submitting it to the PPA for ARM arch support.

Assumptions

Fit 20 or more Texas Instruments (TI) Panda boards into a 4 U rack case with daughter cards that enable secure booting and recovery of the boards when a build is complete.

  • Use a 4U custom rack case that can hold at least 20 Panda boards stacked on their sides.
  • Use an off the shelf 5V DC regulated power supply of sufficient size to power all boards, no individual power supplies. (Estimated to be at least 600 Watts.)
  • Use an off the shelf 12V DC regulated power supply of sufficient size to power the serial remote control relay board.
  • Use off the shelf hardware where possible.
  • Where off the shelf hardware is not possible, describe the required hardware such that it can be created with minimal effort. Where possible when working with hardware vendors get a part number assigned to others can just order the part number.
  • Will provide a secure boot method that will allow the build system to run as root but recover the machine securely and make sure that there are no artifacts left on the next deployment of the hardware. Security of the machine is critical.
  • Use a remote boot system that allows external (out of band) control of the hardware, so that restart is always in the data-centers control. Serial relay control.
  • Design a boot system that even on non-commanded reboots the system will recover in a secure fashion.

Design

Original design called for a daughter card to control booting via relay, this fell though due to vendor not getting a working design completed in time. In the mean time USB Booting over the OTG port has been worked on and is a better solution in terms of cost and time. Essentially there will be a 21st Panda in the box that will be the master and supply the bootloader and kernel over the wire with no SD cards installed at all. The 21st Panda will export it's serial, and Ethernet ports for remote control by LP and IS. It will be on it's own subnet so a side effect of this change is that IS can remotely update the kernel image that the 20 builders boot without having to physically go to the data center. Turns out this is a better design then the original daughter card was.

Implementation

This section should describe a plan of action (the "how") to implement the changes discussed. Could include subsections like:

UI Changes

Should cover changes required to the UI, or specific UI that is required to implement this

Code Changes

Code changes should include an overview of what needs to change, and in some cases even the specific details.

Migration

Include:

  • data migration, if any
  • redirects from old URLs to new ones, if any
  • how users will be pointed to the new way of doing things, if necessary.

Test/Demo Plan

It's important that we are able to test new features, and demonstrate them to users. Use this section to describe a short plan that anybody can follow that demonstrates the feature is working. This can then be used during testing, and to show off after release. Please add an entry to http://testcases.qa.ubuntu.com/Coverage/NewFeatures for tracking test coverage.

This need not be added or completed until the specification is nearing beta.

Unresolved issues

This should highlight any issues that should be addressed in further specifications, and not problems with the specification itself; since any specification with problems cannot be approved.

BoF agenda and discussion

Currently we lack support of ARM Public PPA at Launchpad.

Pandaboard is now public available and can be used as our main arm build machine for the next cycle.

Because the lack of a proper virtualization on ARM, the only possible way would be to do native builds on the machine, giving the user a possibility to control it as root.

Requirements:

  • Rootfs recriated on every boot
  • External rootfs (e.g. USB)
  • Safe boot method
  • Should work the same was as other archs for launchpad

Boot:

  • Default boot sequence for panda: USB -> MMC

  • Lack of ethernet support on current u-boot
  • Sd card would be the easier choice, easier to work with and maintain

Hardware:

  • Possibility to build a reliable expansion board with extra mmc
  • Best solution regarding security for launchpad
  • Delay for building the extra board
  • Expansion board can also handle the external disk, by having a sata controller
  • Maybe sdio to sata?

Wiring for serial cable http://www.avocent.com/Services_and_Support/Technical_Support/Cabling_Options/Cabling_Diagrams.aspx#rj45_cyc.db9.st

Security:

  • Board booting from external SD (expansion board)
  • SD is powered down when not needed anymore (like u-boot) and an external power relay guarantee the proper expansion GPIO doesn't get power again
  • Security is moved from software to hardware

Actions:

  • [davidm, cpearson] Talk with TinCanTools (prpplague) and get a proper hw design for the expansion board - DONE

  • [davidm, cpearson] See if it's possible to plug sdio to sata at the expansion board - DONE
  • [ogra] Create the SD img file that should be used to boot (x-loader, u-boot, kernel and initrd)
  • [ogra] Create proper initrd that will erase and create a brand new rootfs over the external disk
  • [ogra] Create minimal arm rootfs that works with all launchpad requirements

CategorySpec

Specs/N/ARM/public-panda-ppa-build-cluster (last edited 2011-05-18 14:13:27 by pool-96-226-231-215)