#Raspberry Pi: Real Time System - Preempt-RT Patching Tutorial for Kernel 4.14.y

The Preempt-RT tutorial article is the most visited post on my blog. Therefore, I decided to update the tutorial to make it cleaner, and to introduce some other possible modifications and tips for solving problems.

Preempt-RT is a popular patch for the Linux kernel to transform Linux into such a real-time operating system. There is another option to get this done, and that's Xenomai (tutorial and performance tests here).

Tiejun Chen, the main responsible for the official Raspberry-Pi-RT branch, has published a video about Preempt-RT here. It includes tips, results and tests.

Getting the Sources

For this tutorial, you need a host computer running Linux. In my case, I am using Ubuntu 18.04 LTS, but the tutorial should work with any version of Linux.

Requirements

To start in a clean way, make a subdirectory, e.g. rpi-kernel under your home directory:

~$ mkdir ~/rpi-kernel
~$ cd ~/rpi-kernel 
~rpi-kernel$ mkdir rt-kernel

The subdirectory rt-kernel is for the compiled files.

Then, clone the following repositories:

~/rpi-kernel$ git clone https://github.com/raspberrypi/linux.git -b rpi-4.14.y-rt
~/rpi-kernel$ git clone https://github.com/raspberrypi/tools.git

The Raspberry PI kernel source will be downloaded to the linux subdirectory (1.5-2 GB) and the Raspberry PI cross-compilers to the tools subdirectory (1 GB).

The tools directory contains several toolchain versions that are needed to compile the kernel sources:

• tools/arm-bcm2708/arm-bcm2708hardfp-linux-gnueabi
• tools/arm-bcm2708/arm-bcm2708-linux-gnueabi
• tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian

Switching the Branch

The kernel source repository includes several kernel versions. If you've cloned the complete linux kernel repository (without the -b argument), you need to switch to the rpi-4.14.y-rt branch to compile the Preempt-RT patched kernel, otherwise you are already on the right branch:

~/rpi-kernel$ cd linux
~/rpi-kernel/linux$ git checkout rpi-4.14.y-rt
Branch 'rpi-4.14.y-rt' set up to track remote branch 'rpi-4.14.y-rt' from 'origin'.
Switched to a new branch 'rpi-4.14.y-rt'

Configuration

Toolchain

You need to set the following variable before starting to configure and/or compile the kernel source:

~/rpi-kernel$ export ARCH=arm
~/rpi-kernel$ export CROSS_COMPILE=~/rpi-kernel/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin/arm-linux-gnueabihf-
~/rpi-kernel$ export INSTALL_MOD_PATH=~/rpi-kernel/rt-kernel
~/rpi-kernel$ export INSTALL_DTBS_PATH=~/rpi-kernel/rt-kernel

On build systems that support it (x64 - check e.g. cat /proc/version-), you can use the 64-bit cross compiler tools (Rich: Thanks for your suggestion!) and set the environmental variable CROSS_COMPILE with the following:

~/rpi-kernel$ export CROSS_COMPILE=~/rpi-kernel/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin/arm-linux-gnueabihf-

The other env. variables remain as defined above.

The variables set the following:

• ARCH: the system architecture;
• CROSS_COMPILE: the path to the toolchain. If you cloned the tool repository into another directory, you need to change this path.
• INSTALL_MOD_PATH: the path, into where the compiled kernel modules (MOD) are going to be installed.
• INSTALL_DTBS_PATH: the path, into where the compiled DTB (Device Tree Blob) files are going to be installed.

Notes:

• If you decided to clone, or compile the kernel in other directory, you need to modify these paths. Otherwise, you are going to get errors while compiling or configuring the kernel.
• Leave no spaces near the =.
• You can check the variable contain typing e.g. echo $INSTALL_MOD_PATH.
• If you close the terminal or sub-window in Terminator, you need to set the variables again.

Building the Kernel Configuration

Depending on your current Raspberry Pi hardware, you need to set the KERNEL variable and make the right configuration following this:

• Raspberry Pi 1/1.2 B(+), A(+), Zero (W):

  ~/rpi-kernel$ export KERNEL=kernel
  ~/rpi-kernel$ cd ~/rpi-kernel/linux/
  ~/rpi-kernel/linux/$ make bcmrpi_defconfig

• Raspberry Pi 2, 3 B(+):

  ~/rpi-kernel$ export KERNEL=kernel7
  ~/rpi-kernel$ cd ~/rpi-kernel/linux/
  ~/rpi-kernel/linux/$ make bcm2709_defconfig

This is very important, otherwise you are going to waste your time compiling a kernel that is not going to boot on your system.

Check/Modify the Configuration (Optional)

The Preempt-RT branch is configured to compile the patched kernel. Before this branch exists, you needed to configure the kernel after patching the sources. This is not needed anymore, but you can still tune your kernel, or you can check the actual configuration. To do that, type:

~/rpi-kernel/linux/$ make menuconfig

If you get the error make menuconfig' requires the ncurses libraries, install the ncurses library using:

~/rpi-kernel/linux/$ sudo apt-get install libncurses-dev

The most important configurations are:

• Enable CONFIG_PREEMPT_RT_FULL: Kernel Features → Preemption Model (Fully Preemptible Kernel (RT)) → Fully Preemptible Kernel (RT)
• Enable HIGH_RES_TIMERS: General setup → Timers subsystem → High Resolution Timer Support
• Set CONFIG_HZ to 1000Hz (read the notes!): Kernel Features → Timer frequency = 1000 Hz

Notes:

• If you don't find the CONFIG_PREEMPT_RT_FULL option, you didn't switch the branch, and you are still on the master branch.
• CONFIG_PREEMPT_RT_FULL and HIGH_RES_TIMERS are set correctly per default. CONFIG_HZ is set to 100 Hz per default. If you need lower latencies, you can change it to 1000Hz, as I described above. Choosing 1000 Hz instead of 100 Hz will speed up the scheduling interval and things will get more responsive since latencies get lowered. The scheduling interval of the IRQ decreases too (be careful with this).

Compiling the Kernel

To compile the kernel you need to type the following:

~/rpi-kernel/linux$ make -j4 zImage 
~/rpi-kernel/linux$ make -j4 modules 
~/rpi-kernel/linux$ make -j4 dtbs 
~/rpi-kernel/linux$ make -j4 modules_install 
~/rpi-kernel/linux$ make -j4 dtbs_install

Choose the right -jX parameter according to the number of processors that your host computer has. In my case 4.

Take a coffee or may be 2! ;)

Note:

• The last line returned after installing modules_install reports the kernel version that you compiled, e.g.:

  DEPMOD  4.14.52-rt34-v7+

  You are going to need this information for the kernel deployment.

Then, make just a blob of data at the end of the kernel image typing:

~/rpi-kernel/linux$ mkdir $INSTALL_MOD_PATH/boot
~/rpi-kernel/linux$ ./scripts/mkknlimg ./arch/arm/boot/zImage $INSTALL_MOD_PATH/boot/$KERNEL.img

Transfer the Kernel

After the compilation is completed, compress all files to tranfer them to the Raspberry Pi (read the item 5 of "Tips and Solutions" if you want that your Preempt-RT patched kernel lives alongside your normal kernel):

~/rpi-kernel/linux$ cd $INSTALL_MOD_PATH
~/rpi-kernel/rt-kernel$ tar czf ../rt-kernel.tgz *

Then, transfer the resulting '.tgz' file to the Raspberry Pi using scp and your ssh credentials:

~/rpi-kernel/rt-kernel$ cd ..
~/rpi-kernel$ scp rt-kernel.tgz pi@<ipaddress>:/tmp

Change <ipaddress> to the corresponding IP of your Raspberry Pi.

Installing the Kernel Image, Modules & Device Tree Overlays

Before you start doing this, be sure that you've already saved the important data from your Raspberry Pi (may be you should do a MicroSD card backup). The following commands replace the kernel, modules & device tree overlays without making any backup. That means, if it doesn't work because errors ocour, then you are not going to be able to boot your Raspberry Pi as usual. You can get your files from the MicroSD (e.g. connecting to your host computer), but the hardware is not going to boot. Be also aware there could be compatibility issues with some drivers. This tutorial helps you to install the kernel version 4.14.y. Discussion for kernel compatibilities are here.

If you are sure to continue, type the following on the Raspberry Pi:

~$ cd /tmp
/tmp$ tar xzf rt-kernel.tgz
/tmp$ cd boot
/tmp/boot$ sudo cp -rd * /boot/
/tmp/boot$ cd ../lib
/tmp/lib$ sudo cp -dr * /lib/
# thanks for your comment Bart!
/tmp/lib$ cd ../overlays
/tmp/overlays$ sudo cp -d * /boot/overlays
/tmp/overlays$ cd ..
/tmp$ sudo cp -d bcm* /boot/

Add the following entry to /boot/config.txt:

• Raspberry Pi 1/1.2 B(+), A(+), Zero (W):

  # Add the following option:
  kernel=vmlinuz-4.14.52-rt34+

• Raspberry Pi 2, 3 B(+):

  ~$ sudo nano /boot/config.txt
  # Add the following option:
  kernel=vmlinuz-4.14.52-rt34-v7+

  Change the version number corresponding to the kernel version that you've compiled.

Reboot the Raspberry Pi and if all the stars are aligned, you get the Preempt-RT kernel working! I am just kidding, It should work without any problems! ;P. You can test if the kernel is working, typing:

~$ uname -r
4.14.52-rt34-v7+

Isolcpus (Optional)

In multi-processor realtime systems, it is sometimes desirable to isolate some CPUs in the system to enhance their capability to maintain realtime performance. This can be help you to isolate 1 CPU from the regular tasks and assign it an important interruption or task.

1. To isolate a CPU or multiple CPUs, you need to add this line to the boot/cmdline.txt file.

   ~$ sudo nano /boot/cmdline.txt
   # Add one of the following options at the end of the line
   isolcpus=3      # isolate the CPU nr 3
   isolcpus=1,2,3  # isolate the CPUs nr 1, 2 & 3

   • To define the affinity of interruptions to a specific CPU type e.g.:

     ~ $ sudo echo 3 > /proc/irq/62/smp_affinity
     ~ $ sudo echo 3 > /proc/irq/62/smp_affinity_list

     Making a cat of /proc/interrupts to display its contents:

     ~ $ cat /proc/interrupts

     allows seeing which interruptions are getting more scheduled, and which CPU is answering these interruptions.

   • To assign your task to a CPU(s)

     taskset -c 3 python      # only one CPU
     taskset -c 1,2,3 python  # multiple CPUs

     Taskset is used to set or retrieve the CPU affinity of a running process given its PID or to launch a new COMMAND with a given CPU affinity.

Latency Results

The latecy results are improved using the new Preempt-RT patched kernel 4.14.52-rt34-v7+.

Fig. 1a: Latency results 4.14.21-rt17-v7+	Fig. 1b: Latency results 4.14.52-rt34-v7+

The latency results were calculated using the suite RT-Tests, and the test were configured as described here.

Multithread Performance Results

But, the new Preempt-RT patched kernel 4.14.52-rt34-v7+ does not improve the results obtained with the version 4.14.27-rt21-v7+. :(

Fig. 2a: Multi-thread Configuration on Model B+	Fig. 2b: Multi-thread Configuration on Model B+

The performance test results were calculated using the Multi-thread N-Queens Problem solver and configured as described here.

Tips and Solutions

1. This time I left CONFIG_HZ set to 100 Hz (per default), and the kernel booted without any problem, and the Raspberry Pi didn't hang everytime I wanted to download something. If you are having problems with this. This can help you, the problem is the usb interruption:

   • Add the following entries to /boot/cmdline.txt (the patch should resolve this, but may be it doesn't work for you)

     ~$ sudo nano /boot/cmdline.txt
     # Add the following options at the end of the first line:
     dwc_otg.fiq_enable=0 dwc_otg.fiq_fsm_enable=0 dwc_otg.nak_holdoff=0

2. You can also disable the Low Latency Mode (llm) for the MicroSD card:

   ~$ sudo nano /boot/cmdline.txt
   # Add the following option:
   sdhci_bcm2708.enable_llm=0

3. If you get the following error:

   root@ubuntu:/home/user/rpi-kernel/linux# make -j4 zImage
   ./scripts/gcc-version.sh: line 26: /root/rpi-kernel/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin/arm-linux-gnueabihf-gcc: No such file or directory

   you didn't set the variable CROSS_COMPILE to the right path. Or you didn't download the toolchain.

4. Errors like these:

   raspberrypi-firmware soc:firmware: Get Throttled mailbox call failed
   bcm2835_thermal soc:thermal: invalid response
   bcm2835_thermal soc:thermal: Could not get registers: -22

   or

   Timed out waiting for device dev-disk-by\........
   Dependency failed for /boot
   Dependency failed for local File System
   Timed out waiting for device dev-ttyAMA0.device
   you are in emergency mode

   • can be related with the wrong compiled kernel. That means, you compiled a kernel for a Raspberry Pi 3, and you deployed it on Raspberry Pi/Zero, or viceversa.
   • or you need to copy the overlays and bcm files again:

     # copy the following:
     /tmp/lib$ cd ../overlays
     /tmp/overlays$ sudo cp -d * /boot/overlays
     /tmp/overlays$ cd ..
     /tmp$ sudo cp -d bcm* /boot/

     It usually happens when the Raspbian kernel on your SD is version 4.9. (Thanks Bart for your comment!).

5. If you don't want to replace the actual kernel, you can make the following modifications:

   1. rename the new compiled kernel7.img file to e.g. kernel7_rt.img before you gzip the directory,
   2. copy the gzip file to the Raspberry Pi, and install the kernel as usual, but add the following to/boot/config.txt:

      # Add the following option:
      kernel=kernel7_rt.img

      This allows you to boot with the Preempt-RT patched kernel, if you want to go back to your normal kernel, modify the option to kernel=kernel7.img. (Rich: Thanks for your suggestion!).

6. If you get the following error:

   scripts/link-vmlinux.sh: line 61: ~/rpi-kernel/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin/arm-linux-gnueabihf-ar: No such file or directory

   try using the absolute path to the directory and avoid using relative path (~) (Thanks Hans J. for your E-Mails!).