Compcache

Revision 9 as of 2008-07-09 13:40:45

Clear message

Please check the status of this specification in Launchpad before editing it. If it is Approved, contact the Assignee or another knowledgeable person before making changes.

Summary

Compcache provides a possibility of using parts of the memory as virtual compressed swapdevice. We want to include this functionality in Ubuntu Intrepid to lower memory requirements in various setups.

Release Note

Most notably the minimal system requirements will be affected by this, there should be no separate release note for this feature.

Rationale

The RAM requirements for running an ubuntu desktop are currently very high. Especially on LiveCDs and Thin Clients this high demand is noticeable. Additionally there are more and more users with low specced subnotebooks within the ubuntu userbase that have a requirement for being able to run the system on a limited amount of memory. With the Google Summer of Code project [http://code.google.com/p/compcache/ compcache] it is possible to extend the available physical ram with a virtual swapdevice, this technology will be included in ubuntu.

Use Cases

Fred has a laptop with only 256M of ram. He wants to install ubuntu on it and read the online documentation while the install runs from a live session in the background. Fred opens firefox and is able to read the pages of help.ubuntu.com while the install is going on in the background.

Martha is a teacher and just got a bunch of old PCs donated for the class. All of them have 500MHz PII CPUs, 32MB memory, no HDD and PXE boot capable network cards. The principal of the school agreed that money for an LTSP server in in the budget. Marthas husband is Ubuntu enthusiast and just read that with Intrepid Ibex the requirements for LTSP clients dropped to 32MB. On the weekend marthas husband installs Ubuntu LTSP from the alternate CD, sets up the network in her classroom and everything works out of the box.

Design

The compcache module will be loaded and configured as early in the initramfs as possible.

The following parameter will be used to enable compcache in initramfs:

COMPCACHE_SIZE=""

Based on this initramfs.conf value an init-top script derived from the upstream shipped use_compcache.sh will be triggered to do the setup. If COMPCACHE_SIZE is empty, compcache won't get used in the initramfs.

Tools like casper, ltsp or the netbook/mobile image creating software can drop an override file into /usr/share/initramfs-tools/conf.d/ or /etc/initramfs-tools/conf.d/ and run a chrooted update-initramfs to switch the feature on.

Implementation

  • include the kernel module in linux-ubuntu-modules
  • add an initramfs script based on the proof of concept code below:

modprobe -q lzo_compress
modprobe -q lzo_decompress
modprobe tlsf

if [ -n "$COMPCACHE_SIZE" ]; then
    MOD_PARAM="compcache_size_kbytes=$COMPCACHE_SIZE"
fi

modprobe compcache

swapon /dev/ramzswap0 -p 100
  • Since the swapon could happen faster than the device is created (i.e. due to modprobe being slow). A udev rule like below is added to only do the swapon after the device was actually created (prevents us from using sleep calls or wait loops) and teh swapon call will be dropped from the init-top script.

KERNEL=="ramzswap0", ACTION=="add", \\
  RUN+="/sbin/swapon -p 100 /dev/ramzswap0 2>/dev/null"

Related Bugs

Add compcache modules, allowing ubiquity installs on 256MB machines: https://bugs.launchpad.net/ubuntu/+bug/200765

Xubuntu requires more than 128mb of ram to install via LiveCD: https://bugs.edge.launchpad.net/ubuntu/+source/xubuntu-meta/+bug/70561

Outstanding Issues

  • CPU usage has to be reviewed
    • Works on the principle of swap replay; swap in/out events are relayed to userspace, where it is compressed
  • what is the impact on suspend/hibernate/resume ?
    • Apparently there are some known problems http://code.google.com/p/compcache/issues/detail?id=2 talking about freezes on ARM architecture. Since ARM does suspend completely different (the clock is separated from the CPU, so ARM shuts down completely while x86 CPUs have an internal clock and never power down completely on suspend) it might not be an issue at all on x86 based CPUs.

      During testing at UDS no problems with suspend/resume showed up. Nontheless this should see wider user testing OliverGrawert

Test/Demo Plan

  • use the liveCD default session on a machine with 256M
  • use a thin client with 32M

BoF agenda and discussion

  • Compressed memory suffers from inverse diminishing returns; as memory size and processing power become smaller, compressed cache benefits become less significant. For example, a 486DX 66MHz with 16MB of RAM will likely thrash CPU and have to swap a lot anyway. Fortunately, the system is still essentially unusable before we break even; compressed cache for smaller, barely-usable systems is significantly helpful. More interestingly, compressed cache for faster systems with large amounts of memory is extremely significant, as it raises the upper memory bound for physical disk swapping proportionally to the (larger) amount of physical ram while simultaneously imposing less CPU load (and thus performance loss) for the compression operations due to the increased processing power. Microsoft will duplicate this when it goes mainstream in Ubuntu Desktop and Server as a default enabled option; I don't know if Apple will get there first or simply follow suit. --JohnMoser

  • WKdm and WK4x4 come with no license, but work extremely well. Kaplan has proven very approachable but as far as I know he's been busy and never got around to publishing a license statement; I can try to get him to post the original code with either LGPL or BSD/MIT licensing to ubuntu-devel-discuss but I can't comment on his current opinion of this. I haven't been able to contact Wilson. --JohnMoser


CategorySpec