These days the need for custom kernel compiling has almost disappeared. There used to be a time when make-kpkg was almost in daily use, but after the kernel-ppa repository was published, we have found ourselves using either backported Ubuntu kernels or vanilla kernels from kernel-ppa in our systems when a newer kernel was required for some reason.
We run a lot of LTSP thin/fat clients in schools that have some exotic hardware. Some time ago we learnt that some hardware combinations were more prone for crashes than others. After quite a bit of debugging it turned out that the kernel had a bug in USB stack that was triggered by SMART Board drivers. Finding the bug was an interesting exercise itself, but in the process of getting the fix out we needed to update to kernel 3.6 that had a fix in place.
LTSP chroots require aufs/overlayfs support in kernel to work so until now we have been using Ubuntu kernels that have the required patches. 3.6 is not packaged for Ubuntu yet with aufs/overlayfs patches, so we needed to get a vanilla kernel compiled with overlayfs. Compiling the kernel itself is not difficult, but the whole process needed to get it packaged the Debian way was an exercise we wanted to document for others too.
This document describes how to build and package vanilla kernel.org kernels enhanced third-party patches and configured with certain Ubuntu kernel settings and managing the whole thing cleanly in Git. More specifically, this document describes how to
Prepare your working environment for kernel compilation and packaging,
patch v3.6 kernel with OverlayFS,
compile the patched kernel with Ubuntu Quantal’s configuration,
and package the whole thing cleanly in a Debian package to make it ready for distribution.
OverlayFS is required to have a writeable filesystem layer on top of a read-only filesystem layer. In LTSP, the client image is mounted as the base layer and then a writeable layer implemented as a tmpfs mount on top of that. Hence, all modifications to the writeable layer are non-persistent.
This document assumes the reader has basic knowledge on following topics:
Linux command line tools
A kernel obtained from Torvalds’s Git tree.
A kernel obtained from the stable Git tree. Stable kernels are based on latest kernel releases made by Torvalds in his tree and then having various patches applied on top of that.
A kernel obtained from Ubuntu’s release-specific Git tree, for example Quantal’s Git tree. All Ubuntu kernels are based on Stable kernels with various patches (e.g. Ubuntu-specific additions, packaging, cherries picked from elsewhere) applied on top of the stable surface.
Tell Debian packaging tools who you are:
$ DEBFULLNAME="John Doe" $ DEBEMAIL="firstname.lastname@example.org"
This data is used in
debian/changelog and is mandatory part of the
changelog entry. If omitted, packaging tools will come up with
something anyways, so it’s better to set it right, once and for all. I
suggest you to add those to your
~/.bashrc to avoid re-setting those
repeatedly on every session.
Tell Git who you are:
$ git config --global user.name "$DEBFULLNAME" $ git config --global user.email "$DEBEMAIL"
Git uses this data as commit author identity and stores it to your
Create working directory:
$ mkdir -p ~/devel $ cd devel
All build products will be placed in this directory.
Clone the Vanilla kernel repository:
$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git $ cd linux
The default remote is always named origin:
$ git remote origin
However, we are going to have multiple remotes and multiple upstreams (origins), so let’s give a better name for the default remote:
$ git remote rename origin torvalds
Let’s add couple of remotes more, more specifically Ubuntu Quantal’s repository for packaging and configuration stuff and OverlayFS’s repository for, well, OverlayFS:
$ git remote add ubuntu-quantal git://kernel.ubuntu.com/ubuntu/ubuntu-quantal.git $ git remote add overlayfs git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/vfs.git
And then, let’s fetch the data from those remotes:
$ git fetch ubuntu-quantal $ git fetch overlayfs
Now we all set for the next step, patching the kernel.
Patching the Kernel with OverlayFS à la Git
There are several different ways to apply patches to current Git
patch && git commit,
git cherry-pick and
git rebase. Each of these have their own uses:
the good old
patchwhen you don’t have an option,
git amwhen you are applying a series of patches formatted with
git format-patchand probably received via email,
git cherry-pickwhen you want to pick a commit from some other tree,
git rebasewhen you want to take a series of commits and place them on top of another branch (
git rebasesatisifies many other use cases, it’s a true Swiss Army Knife inside Swiss Army Knife, yo dawg).
We are trying to build v3.6 kernel with support for
OverlayFS. OverlayFS has not been merged to the mainline (Torvalds
tree) yet, it’s developed in a separate repository based on Torvalds’s
tree. The goal is to create a tree, which is based on v3.6 and has the
latest version (v15) of OverlayFS on top of it. Luckily,
makes it ridiculously easy. Let’s first create and checkout a new
local branch based on the latest OverlayFS version:
$ git checkout -b v3.6+overlayfs overlayfs/overlayfs.v15
Then rebase the current branch on top of v3.6:
$ git rebase v3.6
Patching the Kernel with Ubuntu stuff
Ubuntu kernel release branches are based on stable kernel releases. In addition to stable version, Ubuntu has different types of changes on top of that:
Ubuntu-specific kernel source modifications: changes in the kernel tree which are not in upstream. Commit messages prefixed with
Ubuntu-specific kernel configuration modifications: changes only in configuration files. Commit messages prefixed with
Debian packaging changes: changes in
debian.master/directories. Commit messages prefixed with
Patches (cherries) picked from upstream: these are not prefixed, but most of them (all?) have a line mentioning where the cherry was picked from and an URL pointing to the corresponding bug report in Launchpad.
We rebased OverlayFS on top of the latest Torvald’s release (v3.6) and
not on top of an older Ubuntu Quantal. However, Ubuntu Quantal’s tree
has all the Debian packaging infrastructre we need. We could have
tried to rebase Ubuntu Quantal’s tree onto v3.6, but it would probably
resulted in lot of conflicts, because, as said earlier, Ubuntu kernel
trees are based on stable releases, but they have also lot of other
changes on top of that. So we decided to do it differently: let’s take
only the necessary bits from Quantal’s tree and apply them on top of
v3.6+overlayfs-branch. In this case, the necessary bits are only
couple of trivial patches (cherries) to
tools/hv, which makes the
whole thing packaging-friendly:
17c9fa8 UBUNTU: tools/hv: add basic manual pages ecb998c UBUNTU: tools/hv: add basic Makefile
Cherries are easy to pick:
$ git cherry-pick -x 5635e65ba0f3f508cdaf290adb42b8d2364c000f $ git cherry-pick -x efea9d22efa602b01d712698f886300a597e2102
That’s it, they should’ve applied cleanly. We are ready for the last step.
Configuration, compilation and packaging:
In Ubuntu kernel build system, the packaging infrastructure lives
debian.master/ directories. Let’s checkout them
to our tree as they are in the latest commit in
$ git checkout ubuntu-quantal/master -- debian $ git checkout ubuntu-quantal/master -- debian.master
The Ubuntu kernel build system tracks ABI changes which consists of tracking exported symbols and modules. The Ubuntu build system enforces also certain configuration otions. Here we are simplifying things a bit and just remove all checks with following simple script:
for i in debian/scripts/*-check debian.master/scripts/*-check do if [ -f "$i" ] then cat - <<EOF >"$i" #!/bin/sh exit 0 EOF chmod 755 "$i" fi done
One more thing to do to satisfy the build system. Let’s just create symbol and module lists:
$ cp debian.master/abi/3.5.0-17.26/i386/generic debian.master/abi/3.5.0-17.26/i386/fatclient $ git add debian.master/abi/3.5.0-17.26/i386/fatclient $ cp debian.master/abi/3.5.0-17.26/i386/generic.modules debian.master/abi/3.5.0-17.26/i386/fatclient.modules $ git add debian.master/abi/3.5.0-17.26/i386/fatclient.modules
Here, we have used 3.5.0-17.26 as the base version, but your version might differ. Change it if necessary.
Then, let’s create a new configuration flavour for LTSP client image:
$ cp /boot/config-$(uname -r) debian.master/config/i386/config.flavour.fatclient $ fakeroot debian/rules clean defaultconfigs $ git add debian.master/config/i386/config.flavour.fatclient
We need to make a couple of changes. Modify
debian.master/etc/getabis to have the following change:
--- debian.master/etc/getabis +++ debian.master/etc/getabis @@ -11,7 +11,7 @@ package_prefixes linux-image linux-image-extra getall armel omap getall armhf omap highbank getall amd64 generic -getall i386 generic +getall i386 generic fatclient # Ports arches and flavours. getall powerpc powerpc-smp powerpc64-smp
debian.master/rules.d/i386.mk according to the following change:
--- debian.master/rules.d/i386.mk +++ debian.master/rules.d/i386.mk @@ -2,7 +2,7 @@ human_arch = 32 bit x86 build_arch = i386 header_arch = x86_64 defconfig = defconfig -flavours = generic +flavours = generic fatclient build_image = bzImage kernel_file = arch/$(build_arch)/boot/bzImage install_file = vmlinuz
We need to create one more file which defines couple of variables for
the build system. Let’s use
$ cp debian.master/control.d/vars.generic debian.master/control.d/vars.fatclient
And then modify it for our needs:
--- debian.master/control.d/vars.generic +++ debian.master/control.d/vars.fatclient @@ -1,6 +1,6 @@ arch="i386 amd64" -supported="Generic" -target="Geared toward desktop and server systems." +supported="Fatclient" +target="Geared toward LTSP client systems." desc="=HUMAN= SMP" bootloader="grub-pc | grub-efi-amd64 | grub-efi-ia32 | grub | lilo (>= 19.1)" provides="kvm-api-4, redhat-cluster-modules, ivtv-modules, ndiswrapper-modules-1.9"
Then, feel free to modify the kernel configuration further:
$ debian/rules editconfigs
Finally, when everything is ready, add new change log entry:
$ dch -v 3.6-999.$(date +%s) --distribution quantal --package linux -c debian.master/changelog Vanilla kernel v3.6
And commit all changes to Git:
$ git commit --amend -a -m "OPINSYS: [Config] New configuration flavour for i386 fatclients"
We are using same kind of message prefixing scheme as Ubuntu does, with the minor execption that UBUNTU is substituted with OPINSYS.
Once the everything is configured, the actual compilation and packaging phase is really simple. We just clean the repository, checkout a new working branch because the build process makes some changes to tracked files and then just make all necessary targets:
$ git clean -fdx $ git checkout -b kernel-build $ fakeroot debian/rules clean $ fakeroot debian/rules binary-indep $ fakeroot debian/rules binary-perarch $ fakeroot debian/rules binary-fatclient
It’s evident that packaging custom kernels is not the most simplest thing to do. However, many of the steps covered in this document need to be executed only once and many more could be automated. For occasional packaging, the ad-hoc process described above might be good enough, but if custom kernel packaging becomes a common practice, custom packaging infrastructure come into a question. For starters, see Linaro’s packaging spells.