LustreInUbuntu

Summary

Lustre is a high-performance GPLed cluster filesystem from clusterfs.com.

Rationale

There are several use cases where you want a scalable shared filesystem with high through-put performance and POSIX:y semantics. Lustre is at the moment the most interesting one, and shipping Ubuntu with or ready for running Lustre would be awesome for that set of Ubuntu users that need it.

Use cases

  • University U has an Ubuntu compute cluster that runs data-intensive computations. Currently their NFS server is a bottleneck, but with Lustre they could scale the network filesystem bandwidth to handle demand.
  • Company C has a service that delivers lots of large files over the network to the public. In decoupling storage with web serving through a shared Lustre setup, frontends and disk backends can be scaled separately according to need.
  • Lab L has a large multiple-cluster setup already running Lustre. Since Ubuntu is much more awesome their current Linux distribution, they want to change. Native support for Lustre in Ubuntu will certainly help.
  • National Laboratory A needs a fault tolerant, scalable, relatively high performance general purpose file system at the core of a variety of file system services. Everything from LAMP farms to Linux on the Desktop. IBM does not support GPFS on Ubuntu, and Lustre is the most interesting of the open source products.

Design

Lustre has three parts, clients (mounting the filesystem), a metadata server (keeping track of what's where) and object storage targets (OSTs, where the data is stored).

Implementation

  • Evaluate current code
  • Merge with Ubuntu kernel
    • If the current code does not apply for the Ubuntu kernel, "patchless" client support might be possible (at a performance cost).
    • Mainline inclusion is scheduled for 1.8, a version planned to release in 2007.
  • Package the userland tools

Code

  • Verify that Lustre can keep up and work with current Ubuntu kernels

Outstanding issues

  • How far does the on-disk format change compatibility stretch? Clusterfs promises compatibility between adjacent releases, how will upgrading after 5 years look?

BoF agenda and discussion

CategorySpec

LustreInUbuntu (last edited 2008-08-06 16:23:19 by localhost)