ProfileMemoryManagementEfficiency
NOTE: This page is part of the Ubuntu Specification process. Please check the status and details in Launchpad before editing. If the spec is Approved then you should contact the Assignee, or another knowledgeable person, before making changes.
Launchpad entry: none yet
Packages affected:
Summary
This spec describes profiling the behavior of glibc's memory allocator and determining if a substitute is needed.
Rationale
The memory allocator responsible for handling malloc() and free() calls is not perfectly 1:1 with the system. More memory is always allocated than actually requested by the programmer. If there is a LOT more held from the system than in use by the program, a new memory allocator design may be warranted.
Use cases
Use cases can be determined after profiling; they will include that memory usage is lowered by a given ratio.
Scope
The scope is a number of programs and allocation patterns; although the target exactly is the memory allocator in glibc. Profiling of alternative allocators like Boehm GC; BSD's libc allocator; or Hoard may also be interesting.
We are most interested in memory waste. Waste is the difference in the amount of memory taken from the system for a program's malloc()s and the actual amount of memory the program requested via malloc(). A small percentage of waste can be attributed to the allocator's management data; however, variations over a large range should not occur. If a program starts with 10% waste and then reaches 60% waste, something is wrong.
Design
Documentation at http://www.dent.med.uni-muenchen.de/~wmglo/malloc-slides.html gives link to a memory allocator profiling tool at ftp://ftp.dent.med.uni-muenchen.de/pub/wmglo/, mtrace.
The mtrace tool will be used to gather and analyze statistics on memory usage and waste.
Implementation
The mtrace preload library is slightly buggy, and occasionally corrupts the output file such that trace-test doesn't work. Someone will have to fix this for long-term tests such as multi-day runs of firefox.
Code
Data preservation and migration
Test numbers
A few preliminary test numbers on mozilla-thunderbird are listed here. First, after just starting it, once the window is drawn and the mailboxes are loaded.
# current size allocated: 14559919 bytes # maximum size allocated: 37143435 bytes # average size allocated: 11150248 bytes # time elapsed : 13.0165 sec # max. sbrk()'ed plus mmap()'ed memory: 38264832 bytes # heap/mem waste at peak: 2.93 % # average heap/mem waste: 19.68 %
Next, after reading 3 messages. I settled on a short message.
# current size allocated: 15394962 bytes # maximum size allocated: 37143435 bytes # average size allocated: 14030679 bytes # time elapsed : 81.1705 sec # max. sbrk()'ed plus mmap()'ed memory: 38264832 bytes # heap/mem waste at peak: 2.93 % # average heap/mem waste: 48.48 %
After browsing ubuntu-devel for a little bit:
# current size allocated: 28886643 bytes # maximum size allocated: 37143435 bytes # average size allocated: 15674257 bytes # time elapsed : 1104.0663 sec # max. sbrk()'ed plus mmap()'ed memory: 38264832 bytes # heap/mem waste at peak: 2.93 % # average heap/mem waste: 50.00 %
The number seems to continue to float up and down around 50%. Notice the peak is less than the maximum size allocated; the difference is due to the housekeeping data the memory allocator produces, the overcommit allocation (allocate more than you need to preempt the program requesting more), and the raw waste from memory fragmentation.
Unresolved issues
BoF agenda and discussion
References
ProfileMemoryManagementEfficiency (last edited 2008-08-06 16:17:34 by localhost)