Compcache

Differences between revisions 3 and 5 (spanning 2 versions)
Revision 3 as of 2008-06-11 22:06:19
Size: 2632
Editor: p5098ed03
Comment:
Revision 5 as of 2008-07-03 01:53:20
Size: 5482
Editor: 199
Comment: Usefulness of CC
Deletions are marked like this. Additions are marked like this.
Line 8: Line 8:
 * '''Packages affected''': linux, udev  * '''Packages affected''': linux-ubuntu-modules, initramfs-tools
Line 12: Line 12:
 Compcache provides a possibility of using parts of the memory as virtual compressed swapdevice. We want to include this functionality in Ubuntu Intrepid to lower memory requirements in various setups. Compcache provides a possibility of using parts of the memory as virtual compressed swapdevice. We want to include this functionality in Ubuntu Intrepid to lower memory requirements in various setups.
Line 16: Line 16:
 Most notably the minimal system requirements will be affected by this, there should be no separate release note for this feature. Most notably the minimal system requirements will be affected by this, there should be no separate release note for this feature.
Line 24: Line 24:
Fred has a laptop with only 256M of ram. He wants to install ubuntu on it and read the online documentation while the install runs from a live session in the background. Fred opens firefox and is able to read the pages of help.ubuntu.com while the install is going on in the background.

Martha is a teacher and just got a bunch of old PCs donated for the class. All of them have 500MHz PII CPUs, 32MB memory, no HDD and PXE boot capable network cards. The principal of the school agreed that money for an LTSP server in in the budget. Marthas husband is Ubuntu enthusiast and just read that with Intrepid Ibex the requirements for LTSP clients dropped to 32MB. On the weekend marthas husband installs Ubuntu LTSP from the alternate CD, sets up the network in her classroom and everything works out of the box.
Line 27: Line 30:
The compcache module will be loaded and configured as early in the initramfs as possible.

The following two parameters will be used to enable compcache in initramfs:

USE_COMPCACHE=True/False

COMPCACHE_SIZE=""

Based on these initramfs.conf values an init-top script derived from the shipped use_compcache.sh will be triggered to do the setup. If COMPCACHE_SIZE is not set the default value (25% of RAM) will be used.

Tools like casper, ltsp or the netbook/mobile image creating software can drop an override file into /usr/share/initramfs-tools/conf.d/ or /etc/initramfs-tools/conf.d/ and run a chrooted update-initramfs to switch the feature on.
Line 30: Line 44:
 * include the kernel module in linux-ubuntu-modules

 * add an initramfs script based on the proof of concept code below:
{{{
modprobe -q lzo_compress
modprobe -q lzo_decompress
modprobe tlsf

if [ -n "$COMPCACHE_SIZE" ]; then
    MOD_PARAM="compcache_size_kbytes=$COMPCACHE_SIZE"
fi

modprobe compcache

swapon /dev/ramzswap0 -p 100
}}}
Line 43: Line 73:
* a cleaner way than the included initscripts should be found and integration
  with udev for the device creation is needed
Line 50: Line 78:

  '''During testing at UDS no problems with suspend/resume showed up. Nontheless this should see wider user testing''' OliverGrawert
Line 58: Line 88:
* Compressed memory suffers from ''inverse'' diminishing returns; as memory size and processing power become smaller, compressed cache benefits become less significant. For example, a 486DX 66MHz with 16MB of RAM will likely thrash CPU and have to swap a lot anyway. Fortunately, the system is still essentially unusable before we break even; compressed cache for smaller, barely-usable systems is significantly helpful. More interestingly, compressed cache for faster systems with large amounts of memory is extremely significant, as it raises the upper memory bound for physical disk swapping proportionally to the (larger) amount of physical ram while simultaneously imposing less CPU load (and thus performance loss) for the compression operations due to the increased processing power. Microsoft will duplicate this when it goes mainstream in Ubuntu Desktop and Server as a default enabled option; I don't know if Apple will get there first or simply follow suit. --JohnMoser

Please check the status of this specification in Launchpad before editing it. If it is Approved, contact the Assignee or another knowledgeable person before making changes.

Summary

Compcache provides a possibility of using parts of the memory as virtual compressed swapdevice. We want to include this functionality in Ubuntu Intrepid to lower memory requirements in various setups.

Release Note

Most notably the minimal system requirements will be affected by this, there should be no separate release note for this feature.

Rationale

The RAM requirements for running an ubuntu desktop are currently very high. Especially on LiveCDs and Thin Clients this high demand is noticeable. Additionally there are more and more users with low specced subnotebooks within the ubuntu userbase that have a requirement for being able to run the system on a limited amount of memory. With the Google Summer of Code project [http://code.google.com/p/compcache/ compcache] it is possible to extend the available physical ram with a virtual swapdevice, this technology will be included in ubuntu.

Use Cases

Fred has a laptop with only 256M of ram. He wants to install ubuntu on it and read the online documentation while the install runs from a live session in the background. Fred opens firefox and is able to read the pages of help.ubuntu.com while the install is going on in the background.

Martha is a teacher and just got a bunch of old PCs donated for the class. All of them have 500MHz PII CPUs, 32MB memory, no HDD and PXE boot capable network cards. The principal of the school agreed that money for an LTSP server in in the budget. Marthas husband is Ubuntu enthusiast and just read that with Intrepid Ibex the requirements for LTSP clients dropped to 32MB. On the weekend marthas husband installs Ubuntu LTSP from the alternate CD, sets up the network in her classroom and everything works out of the box.

Design

The compcache module will be loaded and configured as early in the initramfs as possible.

The following two parameters will be used to enable compcache in initramfs:

USE_COMPCACHE=True/False

COMPCACHE_SIZE=""

Based on these initramfs.conf values an init-top script derived from the shipped use_compcache.sh will be triggered to do the setup. If COMPCACHE_SIZE is not set the default value (25% of RAM) will be used.

Tools like casper, ltsp or the netbook/mobile image creating software can drop an override file into /usr/share/initramfs-tools/conf.d/ or /etc/initramfs-tools/conf.d/ and run a chrooted update-initramfs to switch the feature on.

Implementation

  • include the kernel module in linux-ubuntu-modules
  • add an initramfs script based on the proof of concept code below:

modprobe -q lzo_compress
modprobe -q lzo_decompress
modprobe tlsf

if [ -n "$COMPCACHE_SIZE" ]; then
    MOD_PARAM="compcache_size_kbytes=$COMPCACHE_SIZE"
fi

modprobe compcache

swapon /dev/ramzswap0 -p 100

Related Bugs

Add compcache modules, allowing ubiquity installs on 256MB machines: https://bugs.launchpad.net/ubuntu/+bug/200765

Xubuntu requires more than 128mb of ram to install via LiveCD: https://bugs.edge.launchpad.net/ubuntu/+source/xubuntu-meta/+bug/70561

Outstanding Issues

* CPU usage has to be reviewed

  • - Works on the principle of swap replay; swap in/out events are relayed to userspace, where it is compressed

* what is the impact on suspend/hibernate/resume ?

  • Apparently there are some known problems http://code.google.com/p/compcache/issues/detail?id=2 talking about freezes on ARM architecture. Since ARM does suspend completely different (the clock is separated from the CPU, so ARM shuts down completely while x86 CPUs have an internal clock and never power down completely on suspend) it might not be an issue at all on x86 based CPUs.

    During testing at UDS no problems with suspend/resume showed up. Nontheless this should see wider user testing OliverGrawert

Test/Demo Plan

  • use the liveCD default session on a machine with 256M
  • use a thin client with 32M

BoF agenda and discussion

* Compressed memory suffers from inverse diminishing returns; as memory size and processing power become smaller, compressed cache benefits become less significant. For example, a 486DX 66MHz with 16MB of RAM will likely thrash CPU and have to swap a lot anyway. Fortunately, the system is still essentially unusable before we break even; compressed cache for smaller, barely-usable systems is significantly helpful. More interestingly, compressed cache for faster systems with large amounts of memory is extremely significant, as it raises the upper memory bound for physical disk swapping proportionally to the (larger) amount of physical ram while simultaneously imposing less CPU load (and thus performance loss) for the compression operations due to the increased processing power. Microsoft will duplicate this when it goes mainstream in Ubuntu Desktop and Server as a default enabled option; I don't know if Apple will get there first or simply follow suit. --JohnMoser


CategorySpec

Compcache (last edited 2008-08-06 16:41:47 by localhost)