#Raspberry Pi: Real Time System - Xenomai Patching Tutorial for Kernel 4.14.y
Real Time Systems | 6 min | 22819
Table of contentShowThe website of Xenomai says the following about it Xenomai:
Xenomai brings POSIX and traditional RTOS APIs for porting time-critical applications to Linux-based platforms. When the native Linux kernel cannot meet the response time requirements of the application, Xenomai supplements it with Cobalt, a small real-time infrastructure which schedules time-critical activities independently from the main kernel logic.
Checking the list of supported hardware, the Raspberry Pi 2, 3 B(+) ARM micro (BCM2835) is included.
This tutorial is almost the same as the tutorial for Preempt-RT patching, but in this case, we need to patch the kernel. The patched Preempt-RT sources are included in the official Raspberry Pi repository. If you need a tutorial for patching the kernel click on the link above.
Getting the sources
For this tutorial, you need a host computer running Linux. In my case, I am using Ubuntu 18.04 LTS, but the tutorial should work with any version of Linux.
Requirements
To start in a clean way, make a subdirectory, e.g.
rpi-kernel
under yourhome
directory:~$ mkdir ~/rpi-kernel ~$ cd ~/rpi-kernel ~rpi-kernel$ mkdir rt-kernel
The subdirectory
rt-kernel
is for the compiled files.Then, clone the following repositories:
~/rpi-kernel$ git clone https://github.com/raspberrypi/linux.git ~/rpi-kernel$ git clone https://github.com/raspberrypi/tools.git --depth 3
The Raspberry PI kernel source will be downloaded to the
linux
subdirectory (1-2GB) and the Raspberry PI cross-compilers to thetools
subdirectory (200MB).We need also the Xenomai files and the patch, in this case I took the last stable version (v3.0.7) and the patch for arm32. There is a patch for arm64, but at the moment of writing this, it's still for kernel 4.9. There is also a Xenomai "next" version here, but it is a development version.
~/rpi-kernel$ wget https://xenomai.org/downloads/ipipe/v4.x/arm/ipipe-core-4.14.36-arm-1.patch ~/rpi-kernel$ wget http://xenomai.org/downloads/xenomai/stable/xenomai-3.0.7.tar.bz2 ~/rpi-kernel$ tar xjf xenomai-3.0.7.tar.bz2
Switching the branch
As we are going to patch the kernel source with the Xenomai patch, both versions have to be the same. The Xenomai patches can be downloaded here. At the moment of writing this, the last compatible patch is for kernel version 4.14.36. Looking at the commits in https://github.com/raspberrypi/linux.git there are no merge for this kernel version. We can merge the upstream 4.14.36 and generate a local version for us as:
~/rpi-kernel$ cd linux ~/rpi-kernel/linux$ git reset --hard 719df11c ~/rpi-kernel/linux$ git merge d6949f48093c2d862d9bc39a7a89f2825c55edc4
or we can clone the next kernel version, which is the 4.14.37 and hope that it is still compatible:
~/rpi-kernel$ cd linux ~/rpi-kernel/linux$ git reset --hard 29653ef5475124316b9284adb6cbfc97e9cae48f
I used the second option and it works!
Two files have problems with the patch (also from kernel version 4.14.36):
linux/drivers/irqchip/irq-bcm2835.c linux/drivers/irqchip/irq-bcm2836.c
If you try to prepare the kernel to patch, you are going to get the following error:
Unable to patch kernel 4.14.36/7 with ipipe-core-4.14.36/7-arm-1.patch.
IMPORTANT: But, I included the patched files here. You need to copy them into
linux/drivers/irqchip/
and then everything works as usual. ;)Patching the kernel
To patch the kernel you need to type the following:
~/rpi-kernel$ xenomai-3.0.7/scripts/prepare-kernel.sh --linux=linux/ --arch=arm --ipipe=ipipe-core-4.14.36-arm-1.patch --verbose [ ... ] I-pipe core/arm #1 installed. Links installed. Build system ready.
If you decided to clone, download or compile the kernel in other directories, you need to modify the paths.
Configuration
Tools
You need to set the following variable before starting to configure and/or compile the kernel source.
~/rpi-kernel$ export ARCH=arm ~/rpi-kernel$ export CROSS_COMPILE=~/rpi-kernel/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin/arm-linux-gnueabihf- ~/rpi-kernel$ export INSTALL_MOD_PATH=~/rpi-kernel/rt-kernel ~/rpi-kernel$ export INSTALL_DTBS_PATH=~/rpi-kernel/rt-kernel
Note:
- If you get the following error:
scripts/link-vmlinux.sh: line 61: ~/rpi-kernel/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin/arm-linux-gnueabihf-ar: No such file or directory
try using the absolute path to the directory and avoid using relative path (
~
) (Thanks Hans J. for your E-Mails!).
The variables set the following:
ARCH
: the system architectureCROSS_COMPILE
: the path to the toolchain. If you cloned the tool repository into another directory, you need to change this path.INSTALL_MOD_PATH
: the path, into where the compiled kernel modules (MOD) are going to be installed.INSTALL_DTBS_PATH
: the path, into where the compiled DTB (Device Tree Blob) files are going to be installed.
Notes:
- If you decided to clone, or compile the kernel in other directory, you need to modify these paths. Otherwise, you are going to get some errors while compiling or configuring the kernel.
- Leave no spaces near the
=
. - You can check the variable contain typing e.g.
echo $INSTALL_MOD_PATH
. - If you close the terminal or sub-window in Terminator, you need to set the variables again.
Building the kernel configuration
As I said before, Xenoami is only compatible with Raspberry Pi 2, 3 B(+) -I think-. To build the kernel configuration, we type the following:
~/rpi-kernel$ export KERNEL=kernel7 ~/rpi-kernel$ cd ~/rpi-kernel/linux/ ~/rpi-kernel/linux/$ make bcm2709_defconfig
Then, we need to configure some kernel options, typing the following we get the config menu:
~/rpi-kernel/linux/$ make menuconfig
If you get the error
make menuconfig' requires the ncurses libraries
, install thencurses
library using:~/rpi-kernel/linux/$ sudo apt-get install libncurses-dev
To avoid the warnings you need to disable the following options:
- Disable
CPU Frequency scaling
: CPU Power Management → CPU Frequency scaling → CPU Frequency scaling - Disable
Allow for memory compaction
: Kernel Features → Contiguous Memory Allocator - Disable
Contiguous Memory Allocator
: Kernel Features → Allow for memory compaction - Disable
KGDB: kernel debugger
: KGDB: kernel debugger → Kernel Hacking
Compiling the Kernel
To compile the kernel you need to type the following:
~/rpi-kernel/linux$ make -j4 zImage ~/rpi-kernel/linux$ make -j4 modules ~/rpi-kernel/linux$ make -j4 dtbs ~/rpi-kernel/linux$ make -j4 modules_install ~/rpi-kernel/linux$ make -j4 dtbs_install
Choose the right
-jX
parameter according to the number of processors that your host computer has. In my case 4.Note:
- The last line returned after installing
modules_install
reports the kernel version that you compiled, e.g.:DEPMOD 4.14.37-v7+
Then, make just a blob of data at the end of the kernel image typing:
~/rpi-kernel/linux$ mkdir $INSTALL_MOD_PATH/boot ~/rpi-kernel/linux$ ./scripts/mkknlimg ./arch/arm/boot/zImage $INSTALL_MOD_PATH/boot/$KERNEL.img
Transfer the Kernel
After the compilation is completed, compress all files to tranfer them to the Raspberry Pi:
~/rpi-kernel/linux$ cd $INSTALL_MOD_PATH ~/rpi-kernel/rt-kernel$ tar czf ../xenomai-kernel.tgz *
Then, transfer the resulting '.tgz' file to the Raspberry Pi using
scp
and your ssh credentials:~/rpi-kernel/rt-kernel$ cd .. ~/rpi-kernel$ scp xenomai-kernel.tgz pi@<ipaddress>:/tmp
Change
<ipaddress>
to the corresponding IP of your Raspberry Pi.Installing the Kernel Image, Modules & Device Tree Overlays
Before you start doing this, be sure that you've already saved the important data from your Raspberry Pi (may be you should do a MicroSD card backup). The following commands replace the kernel, modules & device tree overlays without making any backup. That means, if it doesn't work because errors ocour, then you are not going to be able to boot your Raspberry Pi as usual. You can get your files from the MicroSD (e.g. connecting to your host computer), but the hardware is not going to boot. Be also aware there could be compatibility issues with some drivers. This tutorial helps you to install the kernel version 4.14.y. Discussion for kernel compatibilities are here.
If you are sure to continue, type the following on the Raspberry Pi:
~$ cd /tmp /tmp$ tar xzf xenomai-kernel.tgz /tmp$ sudo cp *.dtb /boot/ /tmp$ cd boot /tmp/boot$ sudo cp -rd * /boot/ /tmp/boot$ cd ../lib /tmp/lib$ sudo cp -dr * /lib/ /tmp/lib$ cd ../overlays /tmp/overlays$ sudo cp -d * /boot/overlays /tmp/overlays$ cd .. /tmp$ sudo cp -d bcm* /boot/
Add the following entries to
/boot/config.txt
:kernel=${zImage name} device_tree=bcm2710-rpi-3-b.dtb
and following entries to
/boot/cmdline.txt
. Otherwise the USB drivers are not going to work, and you cannot have neither network nor keyboard!~$ sudo nano /boot/cmdline.txt # Add the following options: dwc_otg.fiq_enable=0 dwc_otg.fiq_fsm_enable=0 dwc_otg.nak_holdoff=0
Some info about this: here.
Reboot the Raspberry Pi and if all the stars are aligned, you get the Xenomai kernel working!
Compiling Xenomai Tools
Xenomani includes a suite of tools for testing its realtime kernel, you can cross-compile these tools typing:
~/rpi-kernel$ cd xenomai-3.0.7 ~/rpi-kernel/xenomai-3.0.7$ ./scripts/bootstrap --with-core=cobalt –enable-debug=partial ~/rpi-kernel/xenomai-3.0.7$ ./configure CFLAGS="-march=armv7-a -mfpu=vfp3" LDFLAGS="-mtune=cortex-a53" --build=i686-pc-linux-gnu --host=arm-linux-gnueabihf --with-core=cobalt --enable-smp CC=${CROSS_COMPILE}gcc LD=${CROSS_COMPILE}ld ~/rpi-kernel/xenomai-3.0.7$ make -j4 install DESTDIR=${PWD}/target
The last line compile the tools, and it could ask you the host root password to install the tools, but don't worry they are going to be installed under the subdirectory
target/
.Again, we compress and transfer the testsuite to the Raspberry Pi using.
~/rpi-kernel/xenomai-3.0.7$ cd target ~/rpi-kernel/xenomai-3.0.7/target$ tar czf ../../xenomai-tools.tgz * ~/rpi-kernel/xenomai-3.0.7$ cd ~/rpi-kernel/ ~/rpi-kernel$ scp xenomai-tools.tgz pi@<ipaddress>:/tmp
Change again
<ipaddress>
to the corresponding IP of your Raspberry Pi.Deploying the Testsuite on Raspberry Pi
Type the following on the Raspberry Pi:
~$ cd /tmp /tmp$ tar xzf xenomai-tools.tgz /tmp$ sudo cp dev/* /dev/ /tmp$ sudo cp usr/* /usr/
You can test the kernel latency using the testsuite:
~$ sudo /usr/xenomai/bin/latency == Sampling period: 1000 us == Test mode: periodic user-mode task == All results in microseconds warming up... RTT| 00:00:01 (periodic user-mode task, 1000 us period, priority 99) RTH|----lat min|----lat avg|----lat max|-overrun|---msw|---lat best|--lat worst RTD| 12.239| 12.693| 19.427| 0| 0| 12.239| 19.427 RTD| 12.291| 12.720| 16.926| 0| 0| 12.239| 19.427 RTD| 12.290| 12.701| 17.447| 0| 0| 12.239| 19.427 RTD| 12.238| 12.709| 14.634| 0| 0| 12.238| 19.427
or the kernel installation using:
~$ sudo /usr/xenomai/bin/xeno-test
and you get a huge report. ;)
Latency Kernel Results
The latency of the Xenomai patched kernel is not as good as I expected! I think, I need to check the kernel configuration again.
Fig. 1a: Latency results Preempt-RT patched Kernel Fig. 1b: Latency results Xenomai patched Kernel The latency results were calculated using the suite RT-Tests, and the test were configured as described here.
- If you get the following error:
We use cookies to improve our services. Read more about how we use cookies and how you can refuse them.
Volodymyr 10.30.2018
I used Linux Linux 4.14.71 (commit 1244bbb3e92135d247e2dddfa6fe5e3e171a9635) with patch https://xenomai.org/downloads/ipipe/v4.x/arm/ipipe-core-4.14.71-arm-4.patch
Volodymyr 10.30.2018
I think that at Tools the following line is cut: export CROSS_COMPILE=~/rpi-kernel/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin/arm-linux-gnueabihf-
Thomas 11.07.2018
Hi, very thanks to your experience. But I have a Pi 3 Model B+ and I try to build the raspberrypi/linux for it. Unfortunately, my kernel7.img was not working on RPi 3 B+. I don't find Xenomai support to Pi 3 Model B+ on the ' https://gitlab.denx.de/Xenomai/xenomai/wikis/Supported_Hardware'. Could you give me some advice? Thank you.
Favereau Marc 11.29.2018
Hi, I have two problems : 1) When i do "/tmp$ sudo cp dev/* /dev/" for xenomail-tools i have the message : "Can't open 'dev/rtp0' for reading: No peripherals or adress" and i can't copy 2) When i do xeno-config --cc i have the path to gcc on the dev computer and not the raspberry path, how can i change it ? I get the last linux kernel and patch : ipipe-core-4.14.71-arm-4.patch
Thanks for four help, Best Regards.
Victoria Tegg 04.29.2019
This is really interesting, You’re a very skilled blogger. I have joined your feed and look forward to seeking more of your great post. Also, I’ve shared your site in my social networks!
Hao 10.15.2019
This tutorial is amazing, yet I am still confused about how to build a cpp project after this. I believe the xenomai library is already built on my linux pc, can I simply include these library?? I have tried to set it in cmake but still failed.......
RM SHAHID 03.01.2020
Hi Sir, I am trying to patch Xenomai on my RPI 3 Model B. I am currently having the version mismatch issue as you mentioned it will occur. You also suggested to copy some file to /linux/drivers/irqchip but I am uncertain which files you were talking about. If you could please reply me on what files specifically I have to copy I will be very much grateful to you.
Tutorial Link:https://lemariva.com/blog/2018/07/raspberry-pi-xenomai-patching-tutorial-for-kernel-4-14-y
amjack 05.30.2020
Hi, it might sound noob, Can I use this step for another hardware instead of RPI. I have Intel i7 NUC.
houssam 06.24.2020
Hello, First I would like to thank you for the great work you've done to create this brilliant tutorial, xenomai works perfectly for me. However, maybe beginners may find some issues, especially when executing make commands. For that :
/usr/bin/arm-linux-gnueabihf-
instead of~/rpi-kernel/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin/arm-linux-gnueabihf-
make clean
command is also a good way to succeed build operation.lscpu
could help to know the number of CPUs our PC have, so the unknown X inmake -jX
could be replaced with the suitable value. That's it, if you find these remarks useful, you can add them to the tutorial.Have a great day and thank you again :) Kind Regards,
Houssam