#Raspberry Pi: Preempt-RT Patching Tutorial for Kernel 4.14.y

There are a lot of tutorials for Raspberry Pi kernel patching using Preempt-RT, but I needed about 15 hours to get it working! I hate Kernel patching! and I have to say, that's not my first time. I have a system working with Ubuntu, RTAI and an Ethercat master. I patched the system more than once. But it is/was always a new thing! and It costs a lot of time! ;)

Then, you have another tutorial here. I only hope that the repo gets official. It could reduce the time to get the RT kernel working, if it is kept updated.

Hardware & Software

Kernel Patch

Preempt-RT is a popular patch for the Linux kernel to transform Linux into a real-time operating system. I patched the standard Raspian kernel with the Preempt-RT Patch and cross-compiled it on my host computer, which is running Ubuntu 16.04 LTS.

Getting the Raspberry Pi Kernel Sources & Patching the Kernel with Preempt-RT

Note: I usually don't include the path (e.g.~/rpi-kernels$) where I type the commands. This time I did it because It makes the reading easier.

Option 1: From Zero to Success! ;)

(Check Option 2, it is easier, but not official! Update 18.03.2018: It is official! Use that option!)

To start the cross-compilation you need to download the latest kernel sources from Github.

~/rpi-kernels$ git clone https://github.com/raspberrypi/linux.git

I switched to the kernel version 4.14.21 using

~/rpi-kernels$ cd linux
~/rpi-kernels/linux$ git checkout d9db059cb982c5478a464c1ff8ce8b17f7768dcc # --> (4.14.21)

The next step is to patch the kernel with the Preempt-RT patch. Be careful, the patch need to match the kernel version. To look for the right patch, open the Makefile located on the kernel sources folder or type head Makefile -n 4 inside the folder and you'll get something like this:

# SPDX-License-Identifier: GPL-2.0
VERSION = 4
PATCHLEVEL = 14
SUBLEVEL = 21

At the time of writing this post, the latest kernel version was 4.14.21. But the latest Preempt-RT patch is for version 4.14.20. You can switch the Linux sources to this version using git checkout 7e83b2ff485cacbf73d27f821e07a8c78ad8cc68, but I tried with the 4.14.21 (d9db059cb982c5478a464c1ff8ce8b17f7768dcc) and It worked (with some changes on the softirq.c file), but It usually doesn't work. ;)

Patches for older kernels can be found in the folder called older, or surfing here.

Download the Preempt-RT patch corresponding to the VERSION, PATCHLEVEL & SUBLEVEL that you have and the usb-dwc_otg-fix patch using:

# Download the Preempt-RT Patch
~/rpi-kernel$ wget https://www.kernel.org/pub/linux/kernel/projects/rt/4.14/patch-4.14.20-rt17.patch.gz 
# if if doesn't work use this:
~/rpi-kernel$ wget https://www.kernel.org/pub/linux/kernel/projects/rt/4.14/older/patch-4.14.20-rt17.patch.gz

# Download a usb-dwc_otg fix
~/rpi-kernel$ wget https://raw.githubusercontent.com/fedberry/kernel/master/usb-dwc_otg-fix-system-lockup-when-interrupts-are-threaded.patch

The usb-dwc_otg patch fix a system lockup when interrupts are threaded, especially the corresponding to the USB.

Note: If the links are not working, copies of these patches are included here.

Patch then the kernel using:

~/rpi-kernel/linux$ zcat ../patch-4.14.20-rt17.patch.gz | patch -p1
# --dry-run: just to see what is going to be patch
~/rpi-kernel/linux$ patch -i ../usb-dwc_otg-fix-system-lockup-when-interrupts-are-threaded.patch -p1 --dry-run 
# continue when all hunks are here and no error is shown
~/rpi-kernel/linux$ patch -i ../usb-dwc_otg-fix-system-lockup-when-interrupts-are-threaded.patch -p1

Check if you have errors on every step. In my case, I got two errors while patching the softirq.c file. I got a softirq.c.rej file in the same folder and I needed to finish the patch using my hands. I included this file, and you can replace the file using:

~/rpi-kernel/linux$ cd kernel
~/rpi-kernel/linux/kernel$ wget https://raw.githubusercontent.com/lemariva/RT-Tools-RPi/master/patched_files/softirq.c

Option 2:

You can download the patched kernel sources from Github from:

##~/rpi-kernel$ git clone https://github.com/TiejunChina/linux.git # update 18.03.2018: old repository##
~/rpi-kernel$ git clone https://github.com/raspberrypi/linux.git # update 18.03.2018: new repository
~/rpi-kernel$ cd linux
~/rpi-kernel/linux$ git checkout rpi-4.14.y-rt  # I tested the version rpi-4.14.21 and it worked.

It includes the usb-dwc_otg-fix (and some other patches?).

Configuring the Tool Chain for Compilation

For cross-compiling the kernel, as I said before I used my host computer running Ubuntu 16.04 LTS. I used the tool chain for ARM downloaded and configured as:

~/rpi-kernel$ git clone https://github.com/raspberrypi/tools.git
~/rpi-kernel$ export ARCH=arm
~/rpi-kernel$ export CROSS_COMPILE=~/rpi-kernel/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin/arm-linux-gnueabihf-
~/rpi-kernel$ export INSTALL_MOD_PATH=~/rpi-kernel/rt-kernel
~/rpi-kernel$ export INSTALL_DTBS_PATH=~/rpi-kernel/rt-kernel

Notes:

• Clone the repo outside the linux folder!
• CROSS_COMPILE should point to arm-linux-gnueabihf- located under tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin/, you are definitely using another system path, then you should change this.
• The directories specified by INSTALL_MOD_PATH and INSTALL_DTBS_PATH define where the modules go after the installation.

Configuring the Kernel

For Raspberry Pi 2/3 Model B, execute these commands inside the linux folder:

~/rpi-kernel/linux$ export KERNEL=kernel7
~/rpi-kernel/linux$ make bcm2709_defconfig

Only as info, for Raspberry Pi Model A(+), B(+), Zero the following commands are needed (not tested Tested: check here.):

~/rpi-kernel/linux$ export KERNEL=kernel
~/rpi-kernel/linux$ make bcmrpi_defconfig

You need to enable and change in the kernel configuration two/three settings:

• CONFIG_PREEMPT_RT_FULL
• HIGH_RES_TIMERS
• CONFIG_HZ: this could improve USB throughput on the RPi (unconfirmed)!

Option 1

You can edit the .config file using

~/rpi-kernel/linux$ nano .config

and search for the options using CTRL+W and set

HIGH_RES_TIMERS=y
CONFIG_PREEMPT_RT_FULL=y
CONFIG_HZ_1000=y
CONFIG_HZ=1000

Do not insert spaces!

Option 2

You can start the menuconfig typing:

~/rpi-kernel/linux$ make menuconfig

and look for the options

• CONFIG_PREEMPT_RT_FULL: Kernel Features → Preemption Model (Fully Preemptible Kernel (RT)) → Fully Preemptible Kernel (RT)
• Set CONFIG_HZ to 1000Hz: Kernel Features → Timer frequency = 1000 Hz
• Enable HIGH_RES_TIMERS: General setup → Timers subsystem → High Resolution Timer Support

After changing these options, don't forget to save the .config file (option Save) and then exit.

Build the Kernel & Transfer to the Raspberry Pi

~/rpi-kernel/linux$ make -j4 zImage 
~/rpi-kernel/linux$ make -j4 modules 
~/rpi-kernel/linux$ make -j4 dtbs 
~/rpi-kernel/linux$ make -j4 modules_install 
~/rpi-kernel/linux$ make -j4 dtbs_install

Choose the right -jX parameter according to the number of processors that your host computer has. In my case 4.

Take a coffee or may be 2! ;).

After the compilation is completed, then compress all files to tranfer them to the Raspberry Pi.

~/rpi-kernel/linux$ mkdir $INSTALL_MOD_PATH/boot
~/rpi-kernel/linux$ ./scripts/mkknlimg ./arch/arm/boot/zImage $INSTALL_MOD_PATH/boot/$KERNEL.img
~/rpi-kernel/linux$ cd $INSTALL_MOD_PATH
~/rpi-kernel/rt-kernel$ tar czf ../rt-kernel.tgz *

Then, transfer the resulting '.tgz' file to the Raspberry Pi using scp and your ssh credentials:

~/rpi-kernel$ scp rt-kernel.tgz pi@<IPAddress>:/tmp

Change <IPAddress> to the corresponding IP of your Raspberry Pi.

Installing the Kernel Image, Modules & Device Tree Overlays

Before you start doing this, be sure that you've already saved the important data from your Raspberry Pi (may be you should do a MicroSD card backup). The following commands replace the kernel, modules & device tree overlays without making any backup. That means, if it doesn't work because errors ocour, then you are not going to be able to boot your Raspberry Pi as usual. You can get your files from the MicroSD (e.g. connecting to your host computer), but it is not going to boot. Be also aware there could be compatibility issues with some drivers. This tutorial helps you to install the kernel version 4.14.21. Discussion for kernel compatibilities are here.

If you are sure to continue, type the following:

~$ cd /tmp
/tmp$ tar xzf rt-kernel.tgz
/tmp$ cd boot
/tmp/boot$ sudo cp -rd * /boot/
/tmp/boot$ cd ../lib
/tmp/lib$ sudo cp -dr * /lib/

Add the following entry to /boot/config.txt:

~$ sudo nano /boot/config.txt
# Add the following option:
## For Raspberry Pi v3 Model B (don't forget to change the kernel version, if you compile another version)
kernel=vmlinuz-4.14.21-rt17-v7+ 
# For Raspberry Pi A(+), B(+), Zero (don't forget to change the kernel version, if you compile another version)
kernel=vmlinuz-4.14.21-rt17+

Change the version number corresponding to the kernel version that you've compiled.

Add the following entries to /boot/cmdline.txt, otherwise it is going to hang everytime you want to download something etc. (the patch should resolve this, but It didn't work!):

~$ sudo nano /boot/cmdline.txt
# Add the following options:
dwc_otg.fiq_enable=0 dwc_otg.fiq_fsm_enable=0 dwc_otg.nak_holdoff=0

You can also disable the Low Latency Mode (llm) for the MicroSD card:

~$ sudo nano /boot/cmdline.txt
# Add the following option:
sdhci_bcm2708.enable_llm=0

After all the changes, reboot your Raspberry Pi:

~$ sudo reboot

If all the stars are aligned, you get the Preempt-RT kernel working! I am just kidding, I should work without problems! ;P. You can test if the kernel is working, typing:

~$ uname -r
4.14.21-rt17-v7

Performance tests between the Standard Raspian kernel and Preempt-RT Patched kernel are available here.

Known Issue (updated 26.02.2018)

Performance!

I found a problem on the patched kernel, and I thing the problem is in every Preempt-RT Raspbian versions. The IRQ/39-dwc_otg process uses more than 30% of the CPU! That's too much! The interruption is related with the USB irq: A known problem of the Preempt-RT patches. I read that the option dwg_otg.speed=1 in /boot/cmdline.txt should mitigate the problem, but It didn't work.

Fig. 3: 3 IRQ/39-dwc_otg processes use more that 30% of the CPU!

If you have another solution to this problem, write a comment!

Download Files

Here you can download the compiled files for Option 1 and Option 2.

Other References

• Raspberry Pi Going Realtime with Preempt-RT
• Raspberry Pi & RT Preempt
• Latency of Raspberry Pi 3 on Standard and Real-Time Linux 4.9 Kernel
• RT-PREEMPT realtime kernel (Raspberry Pi)
• Known Limitations