Ubuntu Wiki

Ubuntu Open Week - Building Upstream Kernels - Leann Ogasawara - Mon, Apr 28, 2008

[20:06] <ogasawara_> ok, lets get started
[20:06] <ogasawara_> Hi Everyone! Welcome to the session on building an upstream kernel.
[20:06] <ogasawara_> My name is Leann Ogasawara and I work for Canonical as a member of the Ubuntu QA Team.
[20:06] <ogasawara_> Lets begin with a little background information on why you'd even want to build the upstream kernel.
[20:07] <ogasawara_> At the beginning of every development cycle, the Ubuntu kernel team rebases the Ubuntu kernel with the upstream vanilla kernel.
[20:07] <ogasawara_> The upstream vanilla kernel is available at http://www.kernel.org and is the kernel maintained by Linus Torvalds.
[20:07] <ogasawara_> At a certain point in the development cycle, the Ubuntu kernel team will cease rebasing with the upstream kernel and focus on stabilizing the Ubuntu kernel code base.
[20:07] <ogasawara_> During this whole development phase, many Ubuntu users are testing and reporting bugs against the actively developed Ubuntu kernel.
[20:08] <ogasawara_> Unfortunately, it is sometimes the case that a bug will exist not only in the Ubuntu kernel code base, but will also exist in the upstream vanilla kernel as well.
[20:08] <ogasawara_> It may also be the case that the bug is already fixed upstream and the fix should be pulled into the Ubuntu kernel.
[20:08] <ogasawara_> Knowing how to build an upstream kernel helps to verify either of these scenarios.
[20:08] <ogasawara_> So lets walk through the steps to build the upstream kernel.
[20:09] <ogasawara_> The actual build process will likely take longer than we time we have allotted here for this session so I'll leave the actual execution of the steps for you to do on your own time.
[20:09] <ogasawara_> Also, please note that the steps we'll be going through is just one way you can go about building the upstream kernel.
[20:09] <ogasawara_> Ok, first you need to install some tools which will help with the process:
[20:09] <ogasawara_> sudo apt-get install git-core kernel-package fakeroot
[20:09] <ogasawara_> Next we need to get the upstream kernel source.
[20:10] <ogasawara_> There are a few different ways to get the upstream kernel source.
[20:10] <ogasawara_> As I had mentioned previously, you could grab the tarball available at http://kernel.org
[20:10] <ogasawara_> Or another option is cloning the kernel git tree - http://git.kernel.org
[20:10] <ogasawara_> We'll clone the git tree in this example:
[20:10] <ogasawara_> git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
[20:10] <ogasawara_> Now we need to configure the kernel.
[20:11] <ogasawara_> Ubuntu has already provided us with a kernel config file that we can use.
[20:11] <ogasawara_> Lets just copy that file into the git tree we cloned:
[20:11] <ogasawara_> cd linux-2.6; cp /boot/config-uname -r .config
[20:11] <ogasawara_> The Ubuntu kernel config file might need some slight updates so run the following and answer any questions that you get prompted for:
[20:11] <ogasawara_> make oldconfig
[20:11] <ogasawara_> Now lets start building the kernel (this will create linux-image and linux-header .deb files):
[20:12] <ogasawara_> fakeroot make-kpkg --initrd --append-to-version=-custom kernel_image kernel_headers
[20:12] <ogasawara_> This will take a while. After this finishes we're ready to install:
[20:12] <ogasawara_> cd ..; sudo dpkg -i linux-image-2.6.25-custom_2.6.25-custom-10.00.Custom_i386.deb linux-headers-2.6.25-custom_2.6.25-custom-10.00.Custom_i386.deb
[20:12] <ogasawara_> Obviously the exact names of the linux-image and linux-header files will vary depending which kernel version you are building.
[20:13] <ogasawara_> Now all you have to do is reboot and choose the new kernel you just built.
[20:13] <ogasawara_> QUESTION: Upstream kernels can be built far faster with multi-core machines. Does make-kpkg offer a way to do a parallel build of a kernel like the lower-level "make -j 5" would (for say, a quad-core)?
[20:14] <ogasawara_> mbt: yes, use CONCURRENCY_LEVEL=N
[20:15] <ogasawara_> CaioAlonso> QUESTION: getting the kernel with git will give you the latest revision (presumably unstable) or the latest stable version?
[20:15] <ogasawara_> CaioAlonso: pulling the kernel with git will get you the latest revision, not necessarily the latest stable version
[20:16] <ogasawara_> ok, moving on (I'll try to field more questions along the way)
[20:16] <ogasawara_> Once you have booted into the kernel, if the bug you're experiencing still exists, that means the bug also exists in the upstream kernel.
[20:16] <ogasawara_> The upstream kernel has its own bug tracking system at http://bugzilla.kernel.org
[20:17] <ogasawara_> It's helpful to also report your bug to the upstream bugzilla.
[20:17] <ogasawara_> It is often the case that once a bug is escalated upstream there is a quick resolution through the help and support of the mainline kernel community.
[20:17] <ogasawara_> Bug reports in Launchpad can also be set up to monitor the upstream bugzilla report.
[20:17] <ogasawara_> It's preferred to have the fix go into the upstream kernel first and then be pulled into the Ubuntu kernel.
[20:18] <ogasawara_> This helps ensure the fix is available to the entire kernel community.
[20:18] <ogasawara_> It's also easier on the Ubuntu kernel team not having to maintain out of tree patches.
[20:18] <ogasawara_> When opening an upstream kernel bugzilla report, there are a few helpful hints to provide the upstream kernel developers with as much information about your bug.
[20:18] <ogasawara_> 1. Verify your bug does not already exist in the upstream bugzilla bugtracking system.
[20:19] <ogasawara_> 2. Include kernel version
[20:19] <ogasawara_> 3. Include your dmesg output
[20:19] <ogasawara_> 4. Include 'sudo lspci -vvnn' output
[20:19] <ogasawara_> 5. Most importantly, if this is a regression, the best information you can provide the kernel developers is the information from doing a git bisect. This will hopefully narrow down the exact patch that is causing the regression. http://www.kernel.org/doc/local/git-quick.html#bisect
[20:19] <ogasawara_> 6. If you've also opened a Launchpad bug report about the bug, you can set up your Launchpad bug report to monitor the upstream bugzilla
[20:19] <ogasawara_> I've documented the entire upstream kernel build process that we just went though at: https://wiki.ubuntu.com/KernelTeam/GitKernelBuild
[20:20] <ogasawara_> That wiki also covers how to file the upstream bug report and also how to link that upstream bug report to the Launchpad bug report.
[20:20] <ogasawara_> Now that you know how to build the upstream vanilla kernel, some of you may be wondering how to build the Ubuntu kernel.
[20:20] <ogasawara_> You can build the Ubuntu kernel using the same exact steps we went through above.
[20:20] <ogasawara_> The Ubuntu kernel git trees can be found at:
[20:21] <ogasawara_> http://kernel.ubuntu.com/git
[20:21] <ogasawara_> Just like we cloned the upstream kernel git tree, you can do the same for the Ubuntu kernel:
[20:21] <ogasawara_> git clone git://kernel.ubuntu.com/ubuntu/ubuntu-intrepid.git
[20:21] <ogasawara_> That will clone the upcoming Intrepid Ibex 8.10 kernel that is currently under development.
[20:21] <ogasawara_> However, the Intrepid kernel is still under going rapid changes so there are no guarantees that it will build cleanly just yet.
[20:21] <ogasawara_> The Ubuntu kernel git trees for previous kernel release for say Dapper, Edgy, Feisty, Gutsy, and Hardy are also still available at http://kernel.ubuntu.com/git
[20:22] <ogasawara_> The Ubuntu kernel source however provides some scripts to hopefully make the above build process we walked through even easier.
[20:22] <ogasawara_> Refer to the following wiki under the "Performing builds" section for more information.
[20:22] <ogasawara_> https://wiki.ubuntu.com/KernelMaintenance
[20:22] <ogasawara_> There is also information there on how to build some of the other kernel related packages such as linux-ubuntu-modules.
[20:22] <ogasawara_> I'll let you go through the steps outlined there on your own time. It should be fairly self explanatory.
[20:24] <ogasawara_> <Flyser> Question: How can I download a stable ubuntu-kernel patchset to patch and build it on my own
[20:24] <ogasawara_> Flyser: I'm not aware of the kernel team releasing an ubuntu-specific patch set specifically
[20:25] <ogasawara_> Flyser: sorry that doesn't really help you much
[20:25] <ogasawara_> Lets get back to why again we're building the upstream and Ubuntu kernels
[20:25] <ogasawara_> We already covered the case where you've confirmed the bug exists upstream as well as in the Ubuntu kernel.
[20:25] <ogasawara_> So what if it's the case that you discover your bug is resolved in the upstream kernel but not in the Ubuntu kernel?
[20:26] <ogasawara_> Many times you may know the exact patch which has been committed upstream and needs to be pulled into the Ubuntu kernel.
[20:26] <ogasawara_> The most useful information you can provide the Ubuntu kernel team will be the upstream git commit id and description.
[20:26] <ogasawara_> The upstream git commit id and description can be found using the following command from within the upstream git kernel tree:
[20:26] <ogasawara_> git log [file|commit id]
[20:26] <ogasawara_> For example:
[20:26] <ogasawara_> ogasawara@emiko:~/linux-2.6$ git log drivers/media/video/cx88/cx88-cards.c
[20:26] <ogasawara_> commit 6b92b3bd7ac91b7e255541f4be9bfd55b12dae41
[20:26] <ogasawara_> Author: Steven Toth <stoth@hauppauge.com>
[20:26] <ogasawara_> Date: Sat Apr 5 16:45:57 2008 -0300
[20:26] <ogasawara_> V4L/DVB (7642): cx88: enable radio GPIO correctly
[20:26] <ogasawara_>
[20:26] <ogasawara_> cx88: enable radio GPIO correctly.
[20:27] <ogasawara_>
[20:27] <ogasawara_> Signed-off-by: Steven Toth <stoth@hauppauge.com>
[20:27] <ogasawara_> Signed-off-by: Mauro Carvalho Chehab <mchehab@infradead.org>
[20:27] <ogasawara_> You can also add the -p flag to view the actual patch associated with that commit:
[20:27] <ogasawara_> git log -p drivers/media/video/cx88/cx88-cards.c
[20:27] <ogasawara_> or
[20:27] <ogasawara_> git log -p 6b92b3bd7ac91b7e255541f4be9bfd55b12dae41
[20:27] <ogasawara_> They should both provide you with the same information.
[20:28] <ogasawara_> Along with proving the upstream git commit id and description, it would be great if you could also confirm the patch does indeed resolve the issue when applied to the Ubuntu kernel source.
[20:28] <ogasawara_> You can save the patch output from running the 'git log -p' command and apply it to the Ubuntu kernel source.
[20:28] <ogasawara_> For those unfamiliar on how to patch the kernel source refer to the patch man page.
[20:28] <ogasawara_> Typically upstream kernel patches can be applied by running the following from the top level directory of the Ubuntu kernel source:
[20:29] <ogasawara_> patch -p1 < [patch.file]
[20:29] <ogasawara_> Then just build the kernel like we walked through above and then test if your bug is fixed.
[20:29] <ogasawara_> You can then open a Launchpad bug report against the "linux" source package which is the source package name for the Ubuntu kernel.
[20:30] <ogasawara_> Make the Title informative but you may also want to preface it with something like [Resolved Upstream].
[20:30] <ogasawara_> In the bug report include a description of the bug you are experiencing.
[20:30] <ogasawara_> Then include the upstream git commit id and description.
[20:30] <ogasawara_> Also if you were able to, be sure to comment that you've applied the upstream patch the Ubuntu kernel and can verify it does resolve the issue you are seeing.
[20:31] <ogasawara_> If you feel the bug is urgent (and please be subjective about this) feel free to then ping me in #ubuntu-bugs or #ubuntu-kernel and I'll try to help get your bug on the radar of the kernel team.
[20:31] <ogasawara_> Otherwise, I regularly go through new kernel bug reports and will likely see it then.
[20:32] <ogasawara_> Now what happens if you know the issue is resolved in the upstream kernel and not in the Ubuntu kernel but you don't know the exact patch which resolved it.
[20:32] <ogasawara_> How do you go about finding the patch?
[20:32] <ogasawara_> you could leverage git bisect to help you narrow down the exact patch. Refer to:
[20:32] <ogasawara_> http://www.kernel.org/doc/local/git-quick.html#bisect
[20:32] <ogasawara_> The only tricky thing is that you will need to reverse the meanings of "good" and "bad" when doing the bisect.
[20:33] <ogasawara_> This is because git bisect assumes that you are searching for the patch that causes the bug not the one that fixes it.
[20:33] <ogasawara_> <pwnguin> QUESTION: I see the word SAUCE a lot in ubuntu changelogs; I assume it means a patch not applied upstream. is it anyone's duty to see that it makes it upstream?
[20:34] <ogasawara_> pwnguin: correct, SAUCE patches are not yet pushed upstream
[20:34] <ogasawara_> pwnguin: however, the kernel team actively submits patches back upstream
[20:35] <ogasawara_> I'd also like to take a moment here to quickly discuss reporting Ubuntu kernel bugs in Launchpad.
[20:35] <ogasawara_> As you can imagine, the Ubuntu kernel receives a rather high volume of bug reports.
[20:35] <ogasawara_> To help streamline the bug reporting process and make the kernel team's life a little easier there are a few things bug reporters can do to help improve reporting bugs:
[20:36] <ogasawara_> 1) Ubuntu kernel bug reports should be filed against the "linux" package. Beginning with Hardy Heron 8.04 the Ubuntu kernel source naming convention changed from linux-source-2.6.xx to just linux.
[20:36] <ogasawara_> 2) Make bug report Titles descriptive - not just "My sound doesn't work" or "Suspend is Broken".
[20:36] <ogasawara_> 3) Make sure bug reports target one specific issue against a specific set of hardware. Many times bugs are specific to the hardware used so even though the symptom may be the same, for ex. failure to Suspend, it really should be separate bug reports. Remember we can easily mark bugs as duplicates later on if necessary.
[20:37] <ogasawara_> 4) As a general rule of thumb, kernel bug reports should at a minimum include the following:
[20:37] <ogasawara_> * cat /proc/version_signature > version.log
[20:37] <ogasawara_> * dmesg > dmesg.log
[20:37] <ogasawara_> * sudo lspci -vvnn > lspci-vvnn.log
[20:37] <ogasawara_> For more information regarding the kernel team bug policy, refer to:
[20:37] <ogasawara_> https://wiki.ubuntu.com/KernelTeamBugPolicies
[20:38] <ogasawara_> That's pretty much all that I wanted to cover here today.
[20:38] <ogasawara_> Are there any other questions people have?
[20:40] <ogasawara_> <[Ramy]> Question: Free software is fun, i want to help, i want your advice about how can i contribute and learn the technology [i'm a computer engineer] while that is not consuming allot of time ?
[20:40] <ogasawara_> so not entirely related to our topic but . . .
[20:41] <ogasawara_> Ramy: if there is a specific area you'd like to become involved in there are usually wikis/mailing lists you can join begin learning about the community you'd want to become involved in
[20:42] <ogasawara_> Ramy: they usually can give you specific advice on how to jump in
[20:42] <ogasawara_> Ramy: specifically for the kernel - http://kernelnewbies.org/ is a good place to start
[20:43] <ogasawara_> <pwnguin> QUESTION: I just completed a git clone from kernel.org, and find that a patch marked SAUCE for hardy is not in upstream, and given my interaction with upstream, seems unlikely that they'd accept the patch as is. Does ubuntu carry patches forward, or will i need to follow up on intrepid in the case that an acceptable fix upstream isn't found quickly?
[20:43] <ogasawara_> pwnguin: patches that do not make it upstream will usually be carried forward from release to release
[20:44] <ogasawara_> <Flyser> QUESTION: (not really on topic but ...) how to get startet with kernel development?
[20:44] <ogasawara_> Flyser: I'd look at the kernel newbies link I posted above
[20:45] <ogasawara_> mnemo> QUESTION: there is many bugs so understanding which ones to focus on is important... how does the ubuntu kernel team figure out what soundcards/networkcards/graphicscards etc work or not work?
[20:45] <ogasawara_> mnemo: many of the developers on the kernel team have access to hardware they can test against, but that is hardly a good coverage of all hardware
[20:46] <ogasawara_> mnemo: that's where the help of the ubuntu community is greatly appreciated - testing and reporting bugs
[20:47] <ogasawara_> <maco> QUESTION: how do you determine if a hardware problem is actually a kernel issue and not a failure in another layer?
[20:47] <ogasawara_> maco: the easiest way to test if it's actually a kernel issue is to try to eliminate the upper layers
[20:48] <ogasawara_> maco: for example, a bug may either be in usplash or the kernel - try booting with the 'quiet' and 'splash' options removed
[20:49] <ogasawara_> <mnemo> QUESTION: Suppose two soundcards dont work, how do you decide which one to work on? Is there a stats page somewhere that says 10k people have card1 and 20k people have card2 etc... I know there is this "hardware testing" program which submits info somewhere but i'm not sure where it ends up exactly?
[20:50] <ogasawara_> mnemo: good question - usually a few things are taken into consideration here. . .
[20:51] <ogasawara_> mnemo: I don't believe there is an official stats page documenting #people who have card1 vs. card2. However, we can use launchpad to help provide some statistics.
[20:52] <ogasawara_> mnemo: we can take a look at the # of duplicates say bug report about card1 has vs card2
[20:52] <ogasawara_> mnemo: we can also take a look at the # of subscribers to each bug report (theoretically the more subscribers the more people are interested in the bug)
[20:53] <ogasawara_> mnemo: and it also may depend if one of our kernel devs has access too one of the cards or not to test
[20:53] <ogasawara_> <maco> QUESTION: if its not something that shows up during boot, if it's something during normal usage, what do you eliminate? hal? And what do you do to test once the upper layers are gone?
[20:54] <ogasawara_> maco: that will typically depend on the symptoms you are seeing
[20:55] <ogasawara_> maco: looking at the logs will hopefully provide any error messages or clues
[20:55] <ogasawara_> <TankEnMate> QUESTION: is there an ubuntu git repository of the kernel?
[20:56] <ogasawara_> TankEnMate: http://kernel.ubuntu.com/git
[20:56] <ogasawara_> ok, we're just about out of time. anything else?
[20:57] <ogasawara_> Ok, thanks everyone. Enjoy the rest of the week!