Raspberry Pi Real Time Systems Preempt-RT Kernel Patching Real-Time Linux Tutorial
#Raspberry Pi: Preempt-RT Patching Tutorial for Kernel 4.14.y (updated: 18.03.2018)
There are a lot of tutorials for Raspberry Pi kernel patching using Preempt-RT, but I needed about 15 hours to get it working! I hate Kernel patching! and I have to say, that's not my first time. I have a system working with Ubuntu, RTAI and an Ethercat master. I patched the system more than once. But it is/was always a new thing! and It costs a lot of time! ;)
Then, you have another tutorial here! I only hope that the repo gets official. It could reduce the time to get the RT kernel working, if it is kept updated.
- Latency Performance
- Kernel CPU & Network Performance
Hardware & Software
|Raspberry Pi 3||x 1|
|Raspbian Lite Image||x 1|
|Raspbian Sources||x 1|
|Kernel RT Patch||x 1|
|Raspbian Sources patched with Preempt-RT (update: 18.03.2018)||x 1|
|Test & Tutorial Data||x 1|
Preempt-RT is a popular patch for the Linux kernel to transform Linux into a real-time operating system. I patched the standard Raspian kernel with the Preempt-RT Patch and cross-compiled it on my host computer, which is running Ubuntu 16.04 LTS.
Getting the Raspberry Pi Kernel Sources & Patching the Kernel with Preempt-RT
Note: I usually don't include the path (e.g.
~/rpi-kernels$) where I type the commands. This time I did it because It makes the reading easier.
Option 1: From Zero to Success! ;) (Check Option 2, it is easier,
but not official! Update 18.03.2018: It is official! Use that option!)
To start the cross-compilation you need to download the latest kernel sources from Github.
~/rpi-kernels$ git clone https://github.com/raspberrypi/linux.git
I switched to the kernel version 4.14.21 using
~/rpi-kernels$ cd linux ~/rpi-kernels/linux$ git checkout d9db059cb982c5478a464c1ff8ce8b17f7768dcc # --> (4.14.21)
The next step is to patch the kernel with the Preempt-RT patch. Be careful, the patch need to match the kernel version. To look for the right patch, open the
Makefile located on the kernel sources folder or type
head Makefile -n 4 inside the folder and you'll get something like this:
# SPDX-License-Identifier: GPL-2.0 VERSION = 4 PATCHLEVEL = 14 SUBLEVEL = 21
At the time of writing this post, the latest kernel version was 4.14.21. But the lastest Preempt-RT patch is for version 4.14.20. You can switch the Linux sources to this version using
git checkout 7e83b2ff485cacbf73d27f821e07a8c78ad8cc68, but I tried with the 4.14.21 (
d9db059cb982c5478a464c1ff8ce8b17f7768dcc) and It worked (with some changes on the
softirq.c file), but It usually doesn't work. ;)
Patches for older kernels can be found in the folder called
older, or surfing here.
Download the Preempt-RT patch corresponding to the VERSION, PATCHLEVEL & SUBLEVEL that you have and the usb-dwc_otg-fix patch using:
# Download the Preempt-RT Patch ~/rpi-kernel$ wget https://www.kernel.org/pub/linux/kernel/projects/rt/4.14/patch-4.14.20-rt17.patch.gz # if if doesn't work use this: ~/rpi-kernel$ wget https://www.kernel.org/pub/linux/kernel/projects/rt/4.14/older/patch-4.14.20-rt17.patch.gz # Download a usb-dwc_otg fix ~/rpi-kernel$ wget https://raw.githubusercontent.com/fedberry/kernel/master/usb-dwc_otg-fix-system-lockup-when-interrupts-are-threaded.patch
The usb-dwc_otg patch fix a system lockup when interrupts are threaded, especially the corresponding to the USB.
Note: If the links are not working, copies of these patches are included here.
Patch then the kernel using:
~/rpi-kernel/linux$ zcat ../patch-4.14.20-rt17.patch.gz | patch -p1 # --dry-run: just to see what is going to be patch ~/rpi-kernel/linux$ patch -i ../usb-dwc_otg-fix-system-lockup-when-interrupts-are-threaded.patch -p1 --dry-run # continue when all hunks are here and no error is shown ~/rpi-kernel/linux$ patch -i ../usb-dwc_otg-fix-system-lockup-when-interrupts-are-threaded.patch -p1
Check if you have errors on every step. In my case, I got two errors while patching the
softirq.c file. I got a
softirq.c.rej file in the same folder and I needed to finish the patch using my hands. I included this file, and you can replace the file using:
~/rpi-kernel/linux$ cd kernel ~/rpi-kernel/linux/kernel$ wget https://raw.githubusercontent.com/lemariva/RT-Tools-RPi/master/patched_files/softirq.c
You can download the patched kernel sources from Github from:
~/rpi-kernel$ git clone https://github.com/TiejunChina/linux.git # update 18.03.2018: old repository ~/rpi-kernel$ git clone https://github.com/raspberrypi/linux.git # update 18.03.2018: new repository ~/rpi-kernel$ cd linux ~/rpi-kernel/linux$ git checkout rpi-4.14.y-rt # I tested the version rpi-4.14.21 and it worked.
It includes the
usb-dwc_otg-fix (and some other patches?).
Configuring the Tool Chain for Compilation
For cross-compiling the kernel, as I said before I used my host computer running Ubuntu 16.04 LTS. I used the tool chain for ARM downloaded and configured as:
~/rpi-kernel$ git clone https://github.com/raspberrypi/tools.git ~/rpi-kernel$ export ARCH=arm ~/rpi-kernel$ export CROSS_COMPILE=~/rpi-kernel/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin/arm-linux-gnueabihf- ~/rpi-kernel$ export INSTALL_MOD_PATH=~/rpi-kernel/rt-kernel ~/rpi-kernel$ export INSTALL_DTBS_PATH=~/rpi-kernel/rt-kernel
- Clone the repo outside the
CROSS_COMPILEshould point to
tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin/, you are definitely using another system path, then you should change this.
- The directories specified by
INSTALL_DTBS_PATHdefine where the modules go after the installation.
Configuring the Kernel
For Raspberry Pi 2/3 Model B, execute these commands inside the
~/rpi-kernel/linux$ export KERNEL=kernel7 ~/rpi-kernel/linux$ make bcm2709_defconfig
Only as info, for Raspberry Pi Model A(+), B(+), Zero the following commands are needed
(not tested Tested: check here.):
~/rpi-kernel/linux$ export KERNEL=kernel ~/rpi-kernel/linux$ make bcmrpi_defconfig
You need to enable and change in the kernel configuration two/three settings:
CONFIG_HZ: this could improve USB throughput on the RPi (unconfirmed)!
You can edit the
.config file using
~/rpi-kernel/linux$ nano .config
and search for the options using
CTRL+W and set
HIGH_RES_TIMERS=y CONFIG_PREEMPT_RT_FULL=y CONFIG_HZ_1000=y CONFIG_HZ=1000
Do not insert spaces!
You can start the
~/rpi-kernel/linux$ make menuconfig
and look for the options
CONFIG_PREEMPT_RT_FULL: Kernel Features → Preemption Model (Fully Preemptible Kernel (RT)) → Fully Preemptible Kernel (RT)
CONFIG_HZto 1000Hz: Kernel Features → Timer frequency = 1000 Hz
HIGH_RES_TIMERS: General setup → Timers subsystem → High Resolution Timer Support
After changing these options, don't forget to save the
.config file (option Save) and then exit.
Build the Kernel & Transfer to the Raspberry Pi
~/rpi-kernel/linux$ make -j4 zImage ~/rpi-kernel/linux$ make -j4 modules ~/rpi-kernel/linux$ make -j4 dtbs ~/rpi-kernel/linux$ make -j4 modules_install ~/rpi-kernel/linux$ make -j4 dtbs_install
Choose the right
-jX parameter according to the number of processors that your host computer has. In my case 4.
Take a coffee or may be 2! ;).
After the compilation is completed, then compress all files to tranfer them to the Raspberry Pi.
~/rpi-kernel/linux$ mkdir $INSTALL_MOD_PATH/boot ~/rpi-kernel/linux$ ./scripts/mkknlimg ./arch/arm/boot/zImage $INSTALL_MOD_PATH/boot/$KERNEL.img ~/rpi-kernel/linux$ cd $INSTALL_MOD_PATH ~/rpi-kernel/rt-kernel$ tar czf ../rt-kernel.tgz *
Then, transfer the resulting '.tgz' file to the Raspberry Pi using
scp and your ssh credentials:
~/rpi-kernel$ scp rt-kernel.tgz pi@<IPAddress>:/tmp
<IPAddress> to the corresponding IP of your Raspberry Pi.
Installing the Kernel Image, Modules & Device Tree Overlays
Before you start doing this, be sure that you've already saved the important data from your Raspberry Pi (may be you should do a MicroSD card backup). The following commands replace the kernel, modules & device tree overlays without making any backup. That means, if it doesn't work because errors ocour, then you are not going to be able to boot your Raspberry Pi as usual. You can get your files from the MicroSD (e.g. connecting to your host computer), but it is not going to boot. Be also aware there could be compatibility issues with some drivers. This tutorial helps you to install the kernel version
4.14.21. Discussion for kernel compatibilities are here.
If you are sure to continue, type the following:
~$ cd /tmp /tmp$ tar xzf rt-kernel.tgz /tmp$ cd boot /tmp/boot$ sudo cp -rd * /boot/ /tmp/boot$ cd ../lib /tmp/lib$ sudo cp -dr * /lib/
Add the following entry to
~$ sudo nano /boot/config.txt # Add the following option: ## For Raspberry Pi v3 Model B (don't forget to change the kernel version, if you compile another version) kernel=vmlinuz-4.14.21-rt17-v7+ # For Raspberry Pi A(+), B(+), Zero (don't forget to change the kernel version, if you compile another version) kernel=vmlinuz-4.14.21-rt17+
Change the version number corresponding to the kernel version that you've compiled.
Add the following entries to
/boot/cmdline.txt, otherwise it is going to hang everytime you want to download something etc. (the patch should resolve this, but It didn't work!):
~$ sudo nano /boot/cmdline.txt # Add the following options: dwc_otg.fiq_enable=0 dwc_otg.fiq_fsm_enable=0 dwc_otg.nak_holdoff=0
You can also disable the Low Latency Mode (llm) for the MicroSD card:
~$ sudo nano /boot/cmdline.txt # Add the following option: sdhci_bcm2708.enable_llm=0
After all the changes, reboot your Raspberry Pi:
~$ sudo reboot
If all the stars are aligned, you get the Preempt-RT kernel working! I am just kidding, I should work without problems! ;P. You can test if the kernel is working, typing:
~$ uname -r 4.14.21-rt17-v7
Performance tests between the Standard Raspian kernel and Preempt-RT Patched kernel are available here.
Known Issue (updated 26.02.2018)
- Latency Performance
- Kernel CPU & Network Performance
I found a problem on the patched kernel, and I thing the problem is in every Preempt-RT Raspbian versions. The IRQ/39-dwc_otg process uses more than 30% of the CPU! That's too much! The interruption is related with the USB irq: A known problem of the Preempt-RT patches. I read that the option
/boot/cmdline.txt should mitigate the problem, but It didn't work.
|Fig. 3: 3 IRQ/39-dwc_otg processes use more that 30% of the CPU!|
If you have another solution to this problem, write a comment!
Here you can download the compiled files for Option 1 and Option 2.