Ubuntu Wiki

• Launchpad Entry: server-lucid-other-cloud-providers

• Created: 2009-11-23

• Contributors: SorenHansen

• Packages affected:

Summary

The cloud efforts in Ubuntu have so far been focused almost exclusively on EC2. We're building images for EC2, and we're offering an EC2 compatble cloud solution, UEC, as well. However, there are many other providers of cloud services in the market place. This spec is about the work we'll undertake to work with or on other clouds.

Release Note

This section should include a paragraph describing the end-user impact of this change. It is meant to be included in the release notes of the first release in which it is implemented. (Not all of these will actually be included in the release notes, at the release manager's discretion; but writing them is a useful exercise.)

It is mandatory.

Rationale

In an effort not to limit ourselves to Amazon's web services, we should try to engage with other cloud providers as well as make it easier to use these other cloud providers' services from Ubuntu.

User stories

1. Johnny has heard of Amazon EC2, the RackSpace cloud, and Linode. He's not sure which is right for him, so he wants to try them out and see what they're like. 1. Peter wants to run application servers on both the RackSpace cloud and EC2 and wants to keep the configuration overhead of this as small as possible. 1. Rackspace Cloud wants to offer Ubuntu images that are officially approved by Ubuntu.

Assumptions

Design

On the client side

http://libcloud.org libcloud offers a simple, consistent Python interface for dealing with a number of different cloud providers. As of this writing (2009-11-24), the following drivers are available:

• EC2
• Slicehost
• Rackspace
• Linode
• VPS.net

• RimuHosting

• GoGrid (there's a driver in their VCS, but the website says it's not supported)

• vCloud
• Terremark

(This list is expected to grow.)

Using libcloud we can make Ubuntu an excellent client for running things on all of these clouds.

On the server side

We seem to be in a reasonably good position. Most (if not all) the cloud providers offer Ubuntu based virtual machines. Only the EC2 images are built by Ubuntu, though. Everyone else's are built by the respective providers themselves.

Ideally, we should be able to generate images that will make every provider happy, and provide a consistent experience for Ubuntu users across all the different clouds.

Most cloud providers should be able to use official Ubuntu kernels, but it is known that at least some of them do not, but rather choose to roll their own kernels for various reasons. Some may need a special config, while other simply are not (understandably) thrilled to use a kernel explicitly labeled "ec2".

The Ubuntu kernel and security team are dedicated to addressing security issues in our kernels, and provide updated kernels for the supported Ubuntu versions in a timely manner. However, since most of the cloud providers seem to use Xen paravirt for VM's, kernel updates usually cannot be done from within the virtual machine. As such, Ubuntu users rely on the providers to not only replace the kernels when there are security updates, but even to build them first.

Implementation

Client side

libcloud is in Debian since 2009-11-18, and will land in Ubuntu shortly (currently sitting in the NEW queue).

For it to be really useful, we should add a simple CLI wrapper around it to expose the functionality to end users.

On the server side

Packages

We should investigate the package selection offered by each of the providers, and see if we can converge on a reasonable package set.

Kernels

We should aim to provide kernel images that are versatile enough that they can be widely used by various cloud providers.

To do this, we should engage with technical contacts at each of them, and gather requirements.

For the cloud providers for whom our kernel images are directly usable, we should instruct them on how to discover security updates and try to get them to commit to applying them in a timely manner.

Test/Demo Plan

It's important that we are able to test new features, and demonstrate them to users. Use this section to describe a short plan that anybody can follow that demonstrates the feature is working. This can then be used during testing, and to show off after release. Please add an entry to http://testcases.qa.ubuntu.com/Coverage/NewFeatures for tracking test coverage.

This need not be added or completed until the specification is nearing beta.

Unresolved issues

This should highlight any issues that should be addressed in further specifications, and not problems with the specification itself; since any specification with problems cannot be approved.

BoF agenda and discussion

Alternative Cloud providers

• see libcloud.org for a good list
• Rackspace
• flexiscale
• gogrid
  • no ubuntu image yet (coming soon)
• slice host
• zinn
• Linnode

libcloud library for using a cloud

• see libcloud.org for supported list

Rackspace was using hardy xen kernels

• hardy kernel doesn't work with latest upstart changes
• Now started using kernel.org kernels
• Also using EC2 kernel renamed
• what customization is being done over server
  • customers want a minimal image
  • packages compiled on customer request
  • compressed server install 90-120MB
    • boot time is fast, copy of image is slow
    • ubuntu minimal server install is closest we have to this need to get recommends down
  • point users at knowledge base and have them install locales
• Kernels are they going to be opened up?
  • plan is to make them part of the image
  • users can upgrade, or bundle their own
• What kind of hooks does rackspace provide for init time configuration
  • create slice, generates password, ip address
  • copy image and mount images copy data in
  • read only user data
  • create a clound server take a snapshot and create new slices off of that
  • xenstore ovf, for windows agent (stuff can be passed in some way)

Upstart script to do init

• present since edgy

What could we put in Ubuntu to make things easier

• libcloud
• various rackspace APIs
• python-cloudfiles
• python-cloudserver
• uploadable images (public, and private), do automatic builds and upload to Rackspace

Rename EC2 kernel so that it can be used anywhere

Networking problem with upstart?

• virtual interfaces can be an issue

How much energy do we want to put into supporting all the different cloud vendors

• EC2 and Rackspace are largest put focus there first

• ZINN? http://src.ielo.net/wiki/niftyname/

• piclound? www.picloud.com/ gateway to EC2 stuff
• Linode

Out images have 2 sets of kernel modules - EC2 and virtual

• we can make them smaller but we want generic images not custom images per cloud
• only EC2 specific part is obtaining of user specific data - want to generalize that
• need real data on what people use so we can produce a smaller image
• apt-get install is a bigger hit than copying over an extra package as part of the image

could rackspace give a different disk with the wanted user data?

• old system probably not, new system probably

Kernel

• missing several modules?

CategorySpec