Building and Packaging Kernels

These days the need for custom kernel compiling has almost disappeared. There used to be a time when make-kpkg was almost in daily use, but after the kernel-ppa repository was published, we have found ourselves using either backported Ubuntu kernels or vanilla kernels from kernel-ppa in our systems when a newer kernel was required for some reason.

We run a lot of LTSP thin/fat clients in schools that have some exotic hardware. Some time ago we learnt that some hardware combinations were more prone for crashes than others. After quite a bit of debugging it turned out that the kernel had a bug in USB stack that was triggered by SMART Board drivers. Finding the bug was an interesting exercise itself, but in the process of getting the fix out we needed to update to kernel 3.6 that had a fix in place.

LTSP chroots require aufs/overlayfs support in kernel to work so until now we have been using Ubuntu kernels that have the required patches. 3.6 is not packaged for Ubuntu yet with aufs/overlayfs patches, so we needed to get a vanilla kernel compiled with overlayfs. Compiling the kernel itself is not difficult, but the whole process needed to get it packaged the Debian way was an exercise we wanted to document for others too.

This document describes how to build and package vanilla kernel.org kernels enhanced third-party patches and configured with certain Ubuntu kernel settings and managing the whole thing cleanly in Git. More specifically, this document describes how to

Prepare your working environment for kernel compilation and packaging,
patch v3.6 kernel with OverlayFS,
compile the patched kernel with Ubuntu Quantal’s configuration,
and package the whole thing cleanly in a Debian package to make it ready for distribution.

OverlayFS is required to have a writeable filesystem layer on top of a read-only filesystem layer. In LTSP, the client image is mounted as the base layer and then a writeable layer implemented as a tmpfs mount on top of that. Hence, all modifications to the writeable layer are non-persistent.

This document assumes the reader has basic knowledge on following topics:

Linux command line tools
Git
Debian packaging

Terminology

Vanilla kernel

A kernel obtained from Torvalds’s Git tree.
Stable kernel

A kernel obtained from the stable Git tree. Stable kernels are based on latest kernel releases made by Torvalds in his tree and then having various patches applied on top of that.
Ubuntu kernel

A kernel obtained from Ubuntu’s release-specific Git tree, for example Quantal’s Git tree. All Ubuntu kernels are based on Stable kernels with various patches (e.g. Ubuntu-specific additions, packaging, cherries picked from elsewhere) applied on top of the stable surface.

Initial Setup

Tell Debian packaging tools who you are:

$ DEBFULLNAME="John Doe"
$ DEBEMAIL="john@doe.tld"

This data is used in debian/changelog and is mandatory part of the changelog entry. If omitted, packaging tools will come up with something anyways, so it’s better to set it right, once and for all. I suggest you to add those to your ~/.bashrc to avoid re-setting those repeatedly on every session.

Tell Git who you are:

$ git config --global user.name "$DEBFULLNAME"
$ git config --global user.email "$DEBEMAIL"

Git uses this data as commit author identity and stores it to your ~/.gitconfig.

Create working directory:

$ mkdir -p ~/devel
$ cd devel

All build products will be placed in this directory.

Clone the Vanilla kernel repository:

$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
$ cd linux

The default remote is always named origin:

$ git remote
origin

However, we are going to have multiple remotes and multiple upstreams (origins), so let’s give a better name for the default remote:

$ git remote rename origin torvalds

Let’s add couple of remotes more, more specifically Ubuntu Quantal’s repository for packaging and configuration stuff and OverlayFS’s repository for, well, OverlayFS:

$ git remote add ubuntu-quantal git://kernel.ubuntu.com/ubuntu/ubuntu-quantal.git
$ git remote add overlayfs git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/vfs.git

And then, let’s fetch the data from those remotes:

$ git fetch ubuntu-quantal
$ git fetch overlayfs

Now we all set for the next step, patching the kernel.

Patching the Kernel with OverlayFS à la Git

There are several different ways to apply patches to current Git branches, i.e. patch && git commit, git am, git cherry-pick and git rebase. Each of these have their own uses:

the good old patch when you don’t have an option,
git am when you are applying a series of patches formatted with git format-patch and probably received via email,
git cherry-pick when you want to pick a commit from some other tree,
and finally git rebase when you want to take a series of commits and place them on top of another branch (git rebase satisifies many other use cases, it’s a true Swiss Army Knife inside Swiss Army Knife, yo dawg).

We are trying to build v3.6 kernel with support for OverlayFS. OverlayFS has not been merged to the mainline (Torvalds tree) yet, it’s developed in a separate repository based on Torvalds’s tree. The goal is to create a tree, which is based on v3.6 and has the latest version (v15) of OverlayFS on top of it. Luckily, git rebase makes it ridiculously easy. Let’s first create and checkout a new local branch based on the latest OverlayFS version:

$ git checkout -b v3.6+overlayfs overlayfs/overlayfs.v15

Then rebase the current branch on top of v3.6:

$ git rebase v3.6

Patching the Kernel with Ubuntu stuff

Ubuntu kernel release branches are based on stable kernel releases. In addition to stable version, Ubuntu has different types of changes on top of that:

Ubuntu-specific kernel source modifications: changes in the kernel tree which are not in upstream. Commit messages prefixed with UBUNTU: SAUCE:.
Ubuntu-specific kernel configuration modifications: changes only in configuration files. Commit messages prefixed with UBUNTU: [Config].
Debian packaging changes: changes in debian/ and debian.master/ directories. Commit messages prefixed with UBUNTU:.
Patches (cherries) picked from upstream: these are not prefixed, but most of them (all?) have a line mentioning where the cherry was picked from and an URL pointing to the corresponding bug report in Launchpad.

We rebased OverlayFS on top of the latest Torvald’s release (v3.6) and not on top of an older Ubuntu Quantal. However, Ubuntu Quantal’s tree has all the Debian packaging infrastructre we need. We could have tried to rebase Ubuntu Quantal’s tree onto v3.6, but it would probably resulted in lot of conflicts, because, as said earlier, Ubuntu kernel trees are based on stable releases, but they have also lot of other changes on top of that. So we decided to do it differently: let’s take only the necessary bits from Quantal’s tree and apply them on top of our v3.6+overlayfs-branch. In this case, the necessary bits are only couple of trivial patches (cherries) to tools/hv, which makes the whole thing packaging-friendly:

  17c9fa8 UBUNTU: tools/hv: add basic manual pages
  ecb998c UBUNTU: tools/hv: add basic Makefile

Cherries are easy to pick:

$ git cherry-pick -x 5635e65ba0f3f508cdaf290adb42b8d2364c000f
$ git cherry-pick -x efea9d22efa602b01d712698f886300a597e2102

That’s it, they should’ve applied cleanly. We are ready for the last step.

Configuration, compilation and packaging:

In Ubuntu kernel build system, the packaging infrastructure lives inside debian/ and debian.master/ directories. Let’s checkout them to our tree as they are in the latest commit in ubuntu-quantal/master:

$ git checkout ubuntu-quantal/master -- debian
$ git checkout ubuntu-quantal/master -- debian.master

The Ubuntu kernel build system tracks ABI changes which consists of tracking exported symbols and modules. The Ubuntu build system enforces also certain configuration otions. Here we are simplifying things a bit and just remove all checks with following simple script:

for i in debian/scripts/*-check debian.master/scripts/*-check
do
    if [ -f "$i" ]
    then
        cat - <<EOF >"$i"
#!/bin/sh
exit 0
EOF
        chmod 755 "$i"
    fi
done

One more thing to do to satisfy the build system. Let’s just create symbol and module lists:

$ cp debian.master/abi/3.5.0-17.26/i386/generic debian.master/abi/3.5.0-17.26/i386/fatclient
$ git add debian.master/abi/3.5.0-17.26/i386/fatclient

$ cp debian.master/abi/3.5.0-17.26/i386/generic.modules debian.master/abi/3.5.0-17.26/i386/fatclient.modules
$ git add debian.master/abi/3.5.0-17.26/i386/fatclient.modules

Here, we have used 3.5.0-17.26 as the base version, but your version might differ. Change it if necessary.

Then, let’s create a new configuration flavour for LTSP client image:

$ cp /boot/config-$(uname -r) debian.master/config/i386/config.flavour.fatclient
$ fakeroot debian/rules clean defaultconfigs
$ git add debian.master/config/i386/config.flavour.fatclient

We need to make a couple of changes. Modify debian.master/etc/getabis to have the following change:

--- debian.master/etc/getabis
+++ debian.master/etc/getabis
@@ -11,7 +11,7 @@
 package_prefixes linux-image linux-image-extra
 getall armel omap
 getall armhf omap highbank
 getall amd64 generic
-getall i386 generic
+getall i386 generic fatclient

 # Ports arches and flavours.
 getall powerpc powerpc-smp powerpc64-smp

And debian.master/rules.d/i386.mk according to the following change:

--- debian.master/rules.d/i386.mk
+++ debian.master/rules.d/i386.mk
@@ -2,7 +2,7 @@
 human_arch     = 32 bit x86
 build_arch     = i386
 header_arch    = x86_64
 defconfig      = defconfig
-flavours       = generic
+flavours       = generic fatclient
 build_image    = bzImage
 kernel_file    = arch/$(build_arch)/boot/bzImage
 install_file   = vmlinuz

We need to create one more file which defines couple of variables for the build system. Let’s use debian.master/control.d/vars.generic as a template:

$ cp debian.master/control.d/vars.generic debian.master/control.d/vars.fatclient

And then modify it for our needs:

--- debian.master/control.d/vars.generic
+++ debian.master/control.d/vars.fatclient
@@ -1,6 +1,6 @@
 arch="i386 amd64"
-supported="Generic"
-target="Geared toward desktop and server systems."
+supported="Fatclient"
+target="Geared toward LTSP client systems."
 desc="=HUMAN= SMP"
 bootloader="grub-pc | grub-efi-amd64 | grub-efi-ia32 | grub | lilo (>= 19.1)"
 provides="kvm-api-4, redhat-cluster-modules, ivtv-modules, ndiswrapper-modules-1.9"

Then, feel free to modify the kernel configuration further:

$ debian/rules editconfigs

Finally, when everything is ready, add new change log entry:

$ dch -v 3.6-999.$(date +%s) --distribution quantal --package linux -c debian.master/changelog Vanilla kernel v3.6

And commit all changes to Git:

$ git commit --amend -a -m "OPINSYS: [Config] New configuration flavour for i386 fatclients"

We are using same kind of message prefixing scheme as Ubuntu does, with the minor execption that UBUNTU is substituted with OPINSYS.

Once the everything is configured, the actual compilation and packaging phase is really simple. We just clean the repository, checkout a new working branch because the build process makes some changes to tracked files and then just make all necessary targets:

$ git clean -fdx
$ git checkout -b kernel-build
$ fakeroot debian/rules clean
$ fakeroot debian/rules binary-indep
$ fakeroot debian/rules binary-perarch
$ fakeroot debian/rules binary-fatclient

Epilogue

It’s evident that packaging custom kernels is not the most simplest thing to do. However, many of the steps covered in this document need to be executed only once and many more could be automated. For occasional packaging, the ad-hoc process described above might be good enough, but if custom kernel packaging becomes a common practice, custom packaging infrastructure come into a question. For starters, see Linaro’s packaging spells.

Happy packaging!

Research and Development