OpenVZ with Archlinux (back in business)

Linux smsitraining 2.6.26.8-OVZ26-gogol.1 #1 SMP Thu Jan 28 15:49:59 PST 2010 x86_64 Intel(R) Xeon(R) CPU E5520 @ 2.27GHz GenuineIntel GNU/Linux

Success! I’ve been working all day on a working OpenVZ kernel on my dev machine. 2.6.18, 2.6.24, 2.6.18rhel5; all borked out on the machine (either mid compile, or crapped out at boot) Anywho I setup a PKGBUILD for 2.6.26 Gogol & made a few modifications, and BAM everything is golden.

I’ve updated git & uploaded the package to AUR: http://aur.archlinux.org/packages.php?ID=34116

BTW, I just found out that an online friend of mine (Wonder) that is heavily involved in ArchLinux has created an Arch template for OpenVZ. Previous to this wonderful contribution OpenVZ has been without an Arch Linux template (that is updated), and it’s been a real drag. You can find the template on the standard templates page:

http://wiki.openvz.org/Download/template/precreated

Kernel Compilation & Arch Linux

Summary:

A brief history of the infamous kernel. Followed by an in depth evaluation on the Arch Linux PKGBUILD of the stock kernel26. In that I’ll delve into the new outline of the PKGBUILD along with ways to customize (.config options, patches, how to prune for performance)

– What is a kernel? –

The kernel is the magical gateway that allows your hardware to communicate with applications. The kernel allows ‘User space’ to call system calls to the ‘Kernel space.’ Such as manipulating a file on a disk (open/close/read/write), or registering CPU commands. This being said the kernel is one of the lowest points of abstraction. It’s the crutch in which the whole of your computer is held up. The kernel manages resources such as CPU, memory, and I/O (disks, peripherals, displays, etc.). The kernel also allows for synchronization and communication between processes. (Inter-Process Communication). It also allows such processes a method to access the aforementioned hardware resources.

The linux kernel is monolithic in nature, yet can be optimized with modules. Modules allow your kernel to be as small as you want, or easily expanded to include everything and the kitchen sink. It also allows you update modules; remove, update, reinsert, and be on your way again. More on modules to follow…

– kernel26-2.6.28 (PKGBUILD) –

Please usher in the all powerful PKGBUILD! (/me welcomes)

Check out the the PKGBUILD in it’s entirety here:

http://repos.archlinux.org/wsvn/packages/kernel26/repos/core-x86_64/PKGBUILD

And for all the other files hit them up on the svn:
http://repos.archlinux.org/wsvn/packages/kernel26/repos/core-x86_64/

For the article, the PKGBUILD will be broken into it’s multiple pieces, and given a little description.

——
1) Common Head
——

  1 # $Id$
  2 # Maintainer: Tobias Powalowski
  3 # Maintainer: Thomas Baechler

  4 pkgname=kernel26                # Build stock -ARCH kernel
  5 # pkgname=kernel26-custom       # Build kernel with a different name
  6 _kernelname=${pkgname#kernel26}
  7 _basekernel=2.6.28
  8 pkgver=${_basekernel}.7
  9 pkgrel=2
 10 _patchname="patch-${pkgver}-${pkgrel}-ARCH"
 11 pkgdesc="The Linux Kernel and modules"
 12 arch=(i686 x86_64)
 13 license=('GPL2')
 14 groups=('base')
 15 url="http://www.kernel.org"
 16 backup=(etc/mkinitcpio.d/${pkgname}.preset)
 17 depends=('coreutils' 'kernel26-firmware>=2.6.28' 'module-init-tools' 'mkinitcpio>=0.5.20')
 18 # pwc, ieee80211 and hostap-driver26 modules are included in kernel26 now
 19 # nforce package support was abandoned by nvidia, kernel modules should cover everything now.
 20 # kernel24 support is dropped since glibc24
 21 replaces=('kernel24' 'kernel24-scsi' 'kernel26-scsi'
 22           'alsa-driver' 'ieee80211' 'hostap-driver26'
 23           'pwc' 'nforce' 'squashfs' 'unionfs' 'ivtv'
 24           'zd1211' 'kvm-modules' 'iwlwifi' 'rt2x00-cvs'
 25           'gspcav1' 'atl2' 'wlan-ng26')
 26 install=kernel26.install
 27 source=(ftp://ftp.kernel.org/pub/linux/kernel/v2.6/linux-$_basekernel.tar.bz2
 28         ftp://ftp.archlinux.org/other/kernel26/${_patchname}.bz2
 29         # the main kernel config files
 30         config config.x86_64
 31         # standard config files for mkinitcpio ramdisk
 32         kernel26.preset)
 33 optdepends=('crda: to set the correct wireless channels of your country')
 34 md5sums=('d351e44709c9810b85e29b877f50968a'
 35          'e535d668afb04a590f03a47356794579'
 36          '3c47674afeae26616acd9dfc4060bd02'
 37          '1f0005febea457f47f26af1726dbf1b2'
 38          '25584700a0a679542929c4bed31433b6')

——

Lets break this down into it’s relative pieces:
4) pkgname (ie. pacman -S kernel26)

6) _kernelname (uname -r)

7) _basekernel (well kernel w/o patches [2.6.28])

pkgver (kernel path version [2.6.28.7])

17) depends (coreutils, module-init-tools, and mkinitcpio are necessary. More on kernel26-firmware later)

26) install* (install file is crucial for mkinitcpio to execute correctly after your kernel has been properly compiled)

Everything else should make sense in this part of the PKGBUILD. If you’re unfamiliar with any of the other variables I would peruse around the arch wiki, and read up on PKGBUILDS

——
2) Make Chunk
——

The next section is where the magic happens. Lines 40-75 start out with a variable to choose the correct .config. Pretend you are your computer (metaphysically) and imagine what you would do if given the following commands. Realize that makepkg does a little magic in the background, yet most everything else is very straightforward.

43) We dive into the extracted kernel directory extracted automagically by makepkg.

46) This is a great place to put patches. More on patches in the next section of the article. After the patches have been applied (or discounted) we move onto configuring the kernel.

Tobias/Thomas are nice and provide a preset .config for x86 & x86_64 (how nice of them).

48-52) determine the machine type and output the correct .config accordingly.

53-55) Check to see if you have ‘_kernelname’ set, and if you do it will change the variable in your .config to match your custom name. ‘2.6.28-ARCH’ is the default for this field.

57-75) Finishes the make. It grabs the correct name of the kernel and pops it into a nifty variable for use later (_kernver). What follows it is a nice block of commented code. The line that follows through just states that whatever the kernel has as defaults will be taken, and the build will commence. Later in the article we will take a look at how to make our own config, and strip out some on the un-necessary things in the kernel. After the config is taken care of, makepkg procedes to build the bzImage (boot image), and the rest of the kernel (modules and stuff)

——

 40 build() {
 41   KARCH=x86
 42
 43   cd ${srcdir}/linux-$_basekernel
 44   # Add -ARCH patches
 45   # See http://projects.archlinux.org/git/?p=linux-2.6-ARCH.git;a=summary
 46   patch -Np1 -i ${srcdir}/${_patchname} || return 1
 47
 48   if [ "$CARCH" = "x86_64" ]; then
 49     cat ../config.x86_64 >./.config
 50   else
 51     cat ../config >./.config
 52   fi
 53   if [ "${_kernelname}" != "" ]; then
 54     sed -i "s|CONFIG_LOCALVERSION=.*|CONFIG_LOCALVERSION=\"${_kernelname}\"|g" ./.config
 55   fi
 56   # get kernel version
 57   make prepare
 58   _kernver="$(make kernelrelease)"
 59   # load configuration
 60   # Configure the kernel. Replace the line below with one of your choice.
 61   #make menuconfig # CLI menu for configuration
 62   #make xconfig # X-based configuration
 63   #make oldconfig # using old config from previous kernel version
 64   # ... or manually edit .config
 65   ####################
 66   # stop here
 67   # this is useful to configure the kernel
 68   #msg "Stopping build"
 69   #return 1
 70   ####################
 71   yes "" | make config
 72   # build!
 73   make bzImage modules || return 1
 74   mkdir -p ${pkgdir}/{lib/modules,boot}
 75   make INSTALL_MOD_PATH=${pkgdir} modules_install || return 1

——
3) The rest
——

76-153) I will refrain from pasting this part of the PKGBUILD. A brief summary of what happens is this:

- Files are copied from working directory to installdir ($pkgdir)

- Miscellaneous files are copied from working dir to install dir.

161-179) This is where we continue:

——

161   # install fallback mkinitcpio.conf file and preset file for kernel
162   install -m644 -D ${srcdir}/kernel26.preset ${pkgdir}/etc/mkinitcpio.d/${pkgname}.preset || return 1
163   # set correct depmod command for install
164   sed \
165     -e  "s/KERNEL_NAME=.*/KERNEL_NAME=${_kernelname}/g" \
166     -e  "s/KERNEL_VERSION=.*/KERNEL_VERSION=${_kernver}/g" \
167     -i $startdir/kernel26.install
168   sed \
169     -e "s|source .*|source /etc/mkinitcpio.d/kernel26${_kernelname}.kver|g" \
170     -e "s|default_image=.*|default_image=\"/boot/${pkgname}.img\"|g" \
171     -e
"s|fallback_image=.*|fallback_image=\"/boot/${pkgname}-fallback.img\"|g" \
172     -i ${pkgdir}/etc/mkinitcpio.d/${pkgname}.preset
173
174   echo -e "# DO NOT EDIT THIS FILE\nALL_kver='${_kernver}'" > ${startdir}/pkg/etc/mkinitcpio.d/${pkgname}.kver
175   # remove unneeded architectures
176   rm -rf ${pkgdir}/usr/src/linux-${_kernver}/arch/{alpha,arm,arm26,avr32,blackfin,cris,frv,h8300,ia64,m32r,m68k,m68knommu,mips,mn10300,parisc,powerpc,ppc,s
390,sh,sh64,sparc,sparc64,um,v850,xtensa}
177   # remove the firmware
178   rm -rf ${pkgdir}/lib/firmware
179 }

——

162) Copies *.preset file (mkinitcpio vars) into install dir. To backstep a little, just in case your unsure what I mean by installdir. Makepkg works with two directories: pkg & src. src is where the package is extracted and compiled. Durring the compile you can specify to compile to the pkg dir, or copy the files over after you have finished your compile (ie. binaries)

164-167) Posts correct variables into your *.install file (uses a simple sed find and replace)

168-172) Pops correct vars into *.preset

174) This is the one line not to screw around with. It creates a .kver file to let mkinitcpio what kernel it’s compiling.

176) Removes erroneous directories

178) I would remove this like if you plan to compile your own kernel. Just leave the firmware in place…

——

– Patches –

Definitely take a peek at:
http://wiki.archlinux.org/index.php/Kernel_Patches_and_Patchsets

This gives a great rundown on patches & patchsets

Other than that, patches are for the tweakers at heart. If you would like to get a fraction more performance benefit then run with patches as opposed to attaching modules. Depending on what you want you might want to patch in the newest version of ext4. Or the latest version of OpenVZ. Or if you would really like to explore, dive into DRBD…

– Modules –

Hooray, Modules! Modules are the easiest way to add functionality to a kernel. Not only do modules reduce the bloat of a once monolithic kernel. They also make it simple to upgrade seperate pieces of the kernel without restarting, or reloading a new kernel. There are certain limits, but they are far and wide, and when you run into one, it’s simple to restart with a new kernel

——
/etc/rc.conf
——

MODULES=(!pcspkr r8169 slhc snd-mixer-oss snd-pcm-oss snd-hwdep
snd-page-alloc snd-pcm snd-timer snd snd-hda-intel soundcore nfs)

——

This is my modules line on my desktop. It’s very simple. The pcsprk is disabled (!), and for the most part the rest of the things loaded are sound drivers.

That’s not to say that I don’t have more modules running. To check the currently running modules on your system run the ‘lsmod’ command. The output will look something similar to this:

——

[fsk141@FSK-Main ~]$ lsmod | more
Module                  Size  Used by
battery                14600  0
nls_cp437               8960  1
vfat                   14464  1
fat                    56760  1 vfat
usb_storage           112832  1
ipv6                  309952  32

——

Hum, battery on a desktop? How absurd, well let’s make sure that it’s not a cmos battery, or anything else important with ‘modinfo’:

——

[fsk141@FSK-Main ~]$ modinfo battery
filename:       /lib/modules/2.6.28-ARCH/kernel/drivers/acpi/battery.ko
license:        GPL
description:    ACPI Battery Driver
author:         Alexey Starikovskiy
author:         Paul Diefenbaugh
alias:          acpi*:PNP0C0A:*
depends:
vermagic:       2.6.28-ARCH SMP preempt mod_unload
parm:           cache_time:cache time in milliseconds (uint)

——

Well, I don’t need ACPI on my desktop, so lets remove it with ‘modprobe -r battery’ You need to run this as root or under sudo… By “removing it” it will remove it from the kernel. Likewise if you were to execute ‘modprobe battery’ it would insert the module.

– Compiling & Tweaking–

To wrap things up lets go through a hypothetical scenario.

1) You edit your PKGBUILD and add the Viper Patchset (Best wishes to the late Vipercinus)

2) You replace ‘yes “” | makeconfig’ with ‘make menuconfig’ (Remove erroneous options, and modules) If you are unsure what something is, then pull up a trusty google page and look it up. For example if you use ext3 as your primary filesystem, then consider removing EXT2, EXT4, XFS, Reiser FS, etc. You might be suprised on how you can cut some time off boot by simply removing pieces from your kernel. (Leave the important stuff, otherwise you’ll just end up with a useless pile of compiled junk)

3) ‘makepkg’ finishes, and you proceded to ‘ls /boot/’ and make sure that your mkinitcpio images were successfully created

4) Add a Grub entry:

——

title  Arch Linux
root   (hd0,0)
kernel /boot/vmlinuz26-custom
root=/dev/sda1 ro
initrd /boot/kernel26-custom.img

——

5) Reboot and cross your fingers. If it reboots, and spits you back into a prompt then you’re golden. Otherwise you will need to troubleshoot what you did wrong, or how you mucked things up.

——

Conclusion:

I wish you the best of luck with your kernel adventures, and Arch Linux in general. The process of building a kernel takes a lot of time to master, and is a very complicated affair. It’s wonderful that the PKGBUILD is able to structure the process so nicely, and allow you create a package with a full kernel, and the instructions to run mkinitcpio, and create the correct boot images. I recommend trying out a couple different patchsets, and trying your hand at making a PKGBUILD of your own kernel. At the very least grab the default kernel, and compile it yourself to see what happens. I can recall the first time I compiled a program, and how confused yet joyous I was. I knew I did something, but I had no idea what that something amounted to. Now after compiling too many programs/kernels to count I look back, and recall the time I had learning. It was troublesome, and there weren’t many resources that were able to help/step you through the whole process. I know that this isn’t an all telling guide for kernel compiling. At the very least, I hope this peaks you interest to learn more.

Enjoy, Jonny Gerold (jonny@fsk141.com)

Last Compile?

I sure hope so… Here is the latest from my Arch Linux PKGBUILD for the OpenVZ 2.6.24 ovz008.1 kernel build. I found what I hope is the last bug, and am just waiting for it to finish (takes 21 minutes total on my 2.0 Ghz Core Duo).

I hope to release 2.6.18, 2.6.24 (WIP), 2.6.26 bulgakov.1/chekhov.1, and 2.6.27 aivazovsky.1 in the future. I’ll let you know when it’s finished.

2.6.18 {i686 – PAE} [OpenVZ *newest* | DRBD 8.3.0

How to compile 2.6.18 with newest DRBD.

It’s kinda a joke that OpenVZ just released a new RHEL5 kernel and it has DRBD 8.2.6 (meh) So I’m doing a short write up on the commands to semi-automatically build a 2.6.18 kernel with OpenVZ & DRBD 8.3.0. I’m choosing to do i686, but if you want to change it to x86_64 or remove PAE then just edit .config & download the correct OpenVZ patch.

mkdir -p /usr/src/kernels/2.6.18_vz_drbd_ibcs/src
cd /usr/src/kernels/2.6.18_vz_drbd_ibcs/src
wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.18.tar.gz
wget http://oss.linbit.com/drbd/8.3/drbd-8.3.0.tar.gz
wget http://voxel.dl.sourceforge.net/sourceforge/linux-abi/ibcs-3_7.tgz
wget http://download.openvz.org/kernel/branches/rhel5-2.6.18/028stab060.2/patches/patch-92.1.18.el5.028stab060.2-combined.gz
wget http://download.openvz.org/kernel/branches/rhel5-2.6.18/028stab060.2/configs/kernel-2.6.18-i686-PAE.config.ovz

tar xzf linux-2.6.18.tar.gz -C ..
tar xzf drbd-8.3.0.tar.gz -C ..
mkdir ../ibcs
tar xzf ibcs-3_7.tgz -C ../ibcs
cp patch-92.1.18.el5.028stab060.2-combined.gz ..
cd ..

gunzip -c patch-92.1.18.el5.028stab060.2-combined.gz | patch -p0 patch-92.1.18.el5.028stab060.2-combined.gz

cd drbd-8.3.0
make KDIR=/usr/src/kernels/2.6.18_vz_drbd_ibcs/linux-2.6.18 kernel-patch
cp patch-linux-2.6.18-drbd-8.3.0 ..
cd ..
patch -p0 < patch-linux-2.6.18-drbd-8.3.0

cd linux-2.6.18
cp ../src/kernel-2.6.18-i686-PAE.config.ovz ./.config
make menuconfig

After you run: make menuconfig, this is the one important thing that you need to change...

Device Drivers > Block devices >  <*>   DRBD Distributed Replicated Block Device support

Then build the kernel:

time make -j7 rpm

Now install, and enjoy

Make the initrd image:

mkinitrd /boot/initrd-2.6.18-PAE.img 2.6.18-prep

Add this to /boot/grub/menu.lst

title CentOS (2.6.18 OPENVZ|DRBD)
        root(hd0,0)
        kernel /vmlinuz-2.6.18-prep ro root=LABEL=/
        initrd /initrd-2.6.18-PAE.img

CentOS 2.6.24 (i686PAE) | OpenVZ ovz006.5 | DRBD 8.3.0 | IBCS 3.7

The goal is to have an i686 kernel with 4GB+ memory support (PAE) with OpenVZ, IBCS, and DRBD.

Environment:
Dell 2900
-8 Cores (1.60Ghz)
-24GB RAM

Operating System:
CentOS 5

Lets start by making ourselves a working directory:

mkdir -p /usr/src/kernels/2.6.24_vz_drbd_ibcs/src
cd /usr/src/kernels/2.6.24_vz_drbd_ibcs/src

Then we need to collect all the parts to build the patched kernel/modules:

wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.24.tar.gz
wget http://oss.linbit.com/drbd/8.3/drbd-8.3.0.tar.gz
wget http://voxel.dl.sourceforge.net/sourceforge/linux-abi/ibcs-3_7.tgz
wget http://download.openvz.org/kernel/branches/2.6.24/2.6.24-ovz006.5/patches/patch-ovz006.5-combined.gz
wget http://download.openvz.org/kernel/branches/2.6.24/2.6.24-ovz006.5/configs/kernel-2.6.24-i686-PAE.config.ovz

Now that we have all the pieces we need lets start extractin & patchin. BTW, I’ll start all the command blocks with a pwd to show you where I am…:

pwd
/usr/src/kernels/2.6.24_vz_drbd_ibcs/src
tar xzf linux-2.6.24.tar.gz -C ..
tar xzf drbd-8.3.0.tar.gz -C ..
mkdir ../ibcs
tar xzf ibcs-3_7.tgz -C ../ibcs
cp patch-ovz006.5-combined.gz ..
cd ..

Ok stay calm, I just put all the source files in /src, and will build the kernel in 2.6.24_vz_drbd_ibcs.

pwd
/usr/src/kernels/2.6.24_vz_drbd_ibcs
gunzip -c patch-ovz006.5-combined.gz | patch -p0 patch-ovz006.5-combined.gz

Lets continue by patching in drbd. You have to actually patch this in as opposed to building it as a module since that doesn’t work.

pwd
/usr/src/kernels/2.6.24_vz_drbd_ibcs
cd drbd-8.3.0
make KDIR=/usr/src/kernels/2.6.24_vz_drbd_ibcs/linux-2.6.24 kernel-patch
cp patch-linux-2.6.24-drbd-8.3.0 ..
cd ..
patch -p0 < patch-linux-2.6.24-drbd-8.3.0

pwd
/usr/src/kernels/2.6.24_vz_drbd_ibcs
cd linux-2.6.24
cp ../src/kernel-2.6.24-i686-PAE.config.ovz ./.config
make menuconfig

After you're dropped into menuconfig you need to set drbd to <*> (to compile drbd into the kernel)

Device Drivers > Block devices >  <*>   DRBD Distributed Replicated Block Device support

At the moment you have a vanilla 2.6.24 kernel with OpenVZ ovz006.5 (patched) and DRBD 8.3.0 RC2 (patched). The only thing left is IBCS which we will install as a module after we have compiled and booted into the OS. As for now lets go ahead and compile, install, and boot.

time make -j7 rpm

The build command 'time make -j7 rpm' will tell you how long it took to build at the end, the -j is how many jobs it can run at once (8 cores - 1 core for OS = 7 cores) and 'rpm' will do package it up for you in and store it at /usr/src/redhat/RPMS{*}

If it compiles correctly you should get no errors, and see something like this:

--output from compile--

real	6m27.445s
user	37m59.198s
sys	6m18.564s

Then navigate to /usr/src/redhat/{athlon, i386, i486, i586, i686, noarch} depending on your archetecture. Mine was in i386.

ls
kernel-2.6.24-2.i386.rpm
pwd
/usr/src/redhat/RPMS/i386
rpm -i kernel-2.6.24-2.i386.rpm

You should get something like the above...

Then you need to make the ramdisk:

mkinitrd /boot/initrd-2.6.24.img 2.6.24

After that add an entry to grub to test your new kernel with OpenVZ and DRBD.

title CentOS (2.6.24 VZ_DRBD_IBCS)
        root(hd0,0)
        kernel /vmlinuz-2.6.24 ro root=LABEL=/
        initrd /initrd-2.6.24.img

If everything goes as expected, then you should boot up. If it doesn't boot, or boots into the wrong kernel then you should check that you did everything correctly... Anywho, you should now be booted into your new kernel.

Lets continue by installing IBCS, and finishing up...

pwd
/usr/src/kernels/2.6.24_vz_drbd_ibcs/ibcs
time make

If you received no errors, then continue edit /etc/rc.local to startup icbs on startup. Add the following to the end of your /etc/rc.local:

/usr/src/kernels/2.6.24_vz_drbd_ibcs/ibcs/abi_ldr

You could move abi_ldr to /usr/bin if you would like, and just make sure to have the path above correspond where you decide to put abi_ldr.

You're Finished!

Resources:
http://www.drbd.org/users-guide/s-build-from-source.html
http://wiki.openvz.org/Download/kernel/2.6.24/2.6.24-ovz006.5
http://www.howtoforge.com/kernel_compilation_centos