• 21
    Jul - 2018

    Real Time Systems
    6 min | 4755

    #Raspberry Pi: Real Time System - Xenomai Patching Tutorial for Kernel 4.14.y

    Real Time Systems | 6 min | 4755


    kernel patching
    raspberry pi
    real time system
    xenomai

    The website of Xenomai says the following about it Xenomai:

    Xenomai brings POSIX and traditional RTOS APIs for porting time-critical applications to Linux-based platforms. When the native Linux kernel cannot meet the response time requirements of the application, Xenomai supplements it with Cobalt, a small real-time infrastructure which schedules time-critical activities independently from the main kernel logic.

    Checking the list of supported hardware, the Raspberry Pi 2, 3 B(+) ARM micro (BCM2835) is included.

    This tutorial is almost the same as the tutorial for Preempt-RT patching, but in this case, we need to patch the kernel. The patched Preempt-RT sources are included in the official Raspberry Pi repository. If you need a tutorial for patching the kernel click on the link above.

    Getting the sources

    For this tutorial, you need a host computer running Linux. In my case, I am using Ubuntu 18.04 LTS, but the tutorial should work with any version of Linux.

    Requirements

    To start in a clean way, make a subdirectory, e.g. rpi-kernel under your home directory:

    ~$ mkdir ~/rpi-kernel
    ~$ cd ~/rpi-kernel 
    ~rpi-kernel$ mkdir rt-kernel

    The subdirectory rt-kernel is for the compiled files.

    Then, clone the following repositories:

    ~/rpi-kernel$ git clone https://github.com/raspberrypi/linux.git
    ~/rpi-kernel$ git clone https://github.com/raspberrypi/tools.git --depth 3

    The Raspberry PI kernel source will be downloaded to the linux subdirectory (1-2GB) and the Raspberry PI cross-compilers to the tools subdirectory (200MB).

    We need also the Xenomai files and the patch, in this case I took the last stable version (v3.0.7) and the patch for arm32. There is a patch for arm64, but at the moment of writing this, it's still for kernel 4.9. There is also a Xenomai "next" version here, but it is a development version.

    ~/rpi-kernel$ wget https://xenomai.org/downloads/ipipe/v4.x/arm/ipipe-core-4.14.36-arm-1.patch
    ~/rpi-kernel$ wget http://xenomai.org/downloads/xenomai/stable/xenomai-3.0.7.tar.bz2
    ~/rpi-kernel$ tar xjf xenomai-3.0.7.tar.bz2

    Switching the branch

    As we are going to patch the kernel source with the Xenomai patch, both versions have to be the same. The Xenomai patches can be downloaded here. At the moment of writing this, the last compatible patch is for kernel version 4.14.36. Looking at the commits in https://github.com/raspberrypi/linux.git there are no merge for this kernel version. We can merge the upstream 4.14.36 and generate a local version for us as:

    ~/rpi-kernel$ cd linux
    ~/rpi-kernel/linux$ git reset --hard 719df11c
    ~/rpi-kernel/linux$ git merge d6949f48093c2d862d9bc39a7a89f2825c55edc4

    or we can clone the next kernel version, which is the 4.14.37 and hope that it is still compatible:

    ~/rpi-kernel$ cd linux
    ~/rpi-kernel/linux$ git reset --hard 29653ef5475124316b9284adb6cbfc97e9cae48f 

    I used the second option and it works!

    Two files have problems with the patch (also from kernel version 4.14.36):

    linux/drivers/irqchip/irq-bcm2835.c
    linux/drivers/irqchip/irq-bcm2836.c

    If you try to prepare the kernel to patch, you are going to get the following error:

    Unable to patch kernel 4.14.36/7 with ipipe-core-4.14.36/7-arm-1.patch.

    IMPORTANT: But, I included the patched files here. You need to copy them into linux/drivers/irqchip/ and then everything works as usual. ;)

    Patching the kernel

    To patch the kernel you need to type the following:

    ~/rpi-kernel$ xenomai-3.0.7/scripts/prepare-kernel.sh --linux=linux/ --arch=arm --ipipe=ipipe-core-4.14.36-arm-1.patch --verbose
    [ ... ]
    I-pipe core/arm #1 installed.
    Links installed.
    Build system ready.

    If you decided to clone, download or compile the kernel in other directories, you need to modify the paths.

    Configuration

    Tools

    You need to set the following variable before starting to configure and/or compile the kernel source.

    ~/rpi-kernel$ export ARCH=arm
    ~/rpi-kernel$ export CROSS_COMPILE=~/rpi-kernel/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin/arm-linux-gnueabihf-
    ~/rpi-kernel$ export INSTALL_MOD_PATH=~/rpi-kernel/rt-kernel
    ~/rpi-kernel$ export INSTALL_DTBS_PATH=~/rpi-kernel/rt-kernel

    Note:

    • If you get the following error:
      scripts/link-vmlinux.sh: line 61: ~/rpi-kernel/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin/arm-linux-gnueabihf-ar: No such file or directory

      try using the absolute path to the directory and avoid using relative path (~) (Thanks Hans J. for your E-Mails!).

    The variables set the following:

    • ARCH: the system architecture
    • CROSS_COMPILE: the path to the toolchain. If you cloned the tool repository into another directory, you need to change this path.
    • INSTALL_MOD_PATH: the path, into where the compiled kernel modules (MOD) are going to be installed.
    • INSTALL_DTBS_PATH: the path, into where the compiled DTB (Device Tree Blob) files are going to be installed.

    Notes:

    • If you decided to clone, or compile the kernel in other directory, you need to modify these paths. Otherwise, you are going to get some errors while compiling or configuring the kernel.
    • Leave no spaces near the =.
    • You can check the variable contain typing e.g. echo $INSTALL_MOD_PATH.
    • If you close the terminal or sub-window in Terminator, you need to set the variables again.

    Building the kernel configuration

    As I said before, Xenoami is only compatible with Raspberry Pi 2, 3 B(+) -I think-. To build the kernel configuration, we type the following:

    ~/rpi-kernel$ export KERNEL=kernel7
    ~/rpi-kernel$ cd ~/rpi-kernel/linux/
    ~/rpi-kernel/linux/$ make bcm2709_defconfig

    Then, we need to configure some kernel options, typing the following we get the config menu:

    ~/rpi-kernel/linux/$ make menuconfig

    If you get the error make menuconfig' requires the ncurses libraries, install the ncurses library using:

    ~/rpi-kernel/linux/$ sudo apt-get install libncurses-dev

    To avoid the warnings you need to disable the following options:

    • Disable CPU Frequency scaling: CPU Power Management → CPU Frequency scaling → CPU Frequency scaling
    • Disable Allow for memory compaction: Kernel Features → Contiguous Memory Allocator
    • Disable Contiguous Memory Allocator: Kernel Features → Allow for memory compaction
    • Disable KGDB: kernel debugger: KGDB: kernel debugger → Kernel Hacking

    Compiling the Kernel

    To compile the kernel you need to type the following:

    ~/rpi-kernel/linux$ make -j4 zImage 
    ~/rpi-kernel/linux$ make -j4 modules 
    ~/rpi-kernel/linux$ make -j4 dtbs 
    ~/rpi-kernel/linux$ make -j4 modules_install 
    ~/rpi-kernel/linux$ make -j4 dtbs_install

    Choose the right -jX parameter according to the number of processors that your host computer has. In my case 4.

    Note:

    • The last line returned after installing modules_install reports the kernel version that you compiled, e.g.:
      DEPMOD  4.14.37-v7+

    Then, make just a blob of data at the end of the kernel image typing:

    ~/rpi-kernel/linux$ mkdir $INSTALL_MOD_PATH/boot
    ~/rpi-kernel/linux$ ./scripts/mkknlimg ./arch/arm/boot/zImage $INSTALL_MOD_PATH/boot/$KERNEL.img

    Transfer the Kernel

    After the compilation is completed, compress all files to tranfer them to the Raspberry Pi:

    ~/rpi-kernel/linux$ cd $INSTALL_MOD_PATH
    ~/rpi-kernel/rt-kernel$ tar czf ../xenomai-kernel.tgz *

    Then, transfer the resulting '.tgz' file to the Raspberry Pi using scp and your ssh credentials:

    ~/rpi-kernel/rt-kernel$ cd ..
    ~/rpi-kernel$ scp xenomai-kernel.tgz pi@<ipaddress>:/tmp

    Change <ipaddress> to the corresponding IP of your Raspberry Pi.

    Installing the Kernel Image, Modules & Device Tree Overlays

    Before you start doing this, be sure that you've already saved the important data from your Raspberry Pi (may be you should do a MicroSD card backup). The following commands replace the kernel, modules & device tree overlays without making any backup. That means, if it doesn't work because errors ocour, then you are not going to be able to boot your Raspberry Pi as usual. You can get your files from the MicroSD (e.g. connecting to your host computer), but the hardware is not going to boot. Be also aware there could be compatibility issues with some drivers. This tutorial helps you to install the kernel version 4.14.y. Discussion for kernel compatibilities are here.

    If you are sure to continue, type the following on the Raspberry Pi:

    ~$ cd /tmp
    /tmp$ tar xzf xenomai-kernel.tgz
    /tmp$ sudo cp *.dtb /boot/
    /tmp$ cd boot
    /tmp/boot$ sudo cp -rd * /boot/
    /tmp/boot$ cd ../lib
    /tmp/lib$ sudo cp -dr * /lib/
    /tmp/lib$ cd ../overlays
    /tmp/overlays$ sudo cp -d * /boot/overlays
    /tmp/overlays$ cd ..
    /tmp$ sudo cp -d bcm* /boot/

    Add the following entries to /boot/config.txt:

    kernel=${zImage name}
    device_tree=bcm2710-rpi-3-b.dtb

    and following entries to /boot/cmdline.txt. Otherwise the USB drivers are not going to work, and you cannot have neither network nor keyboard!

    ~$ sudo nano /boot/cmdline.txt
    # Add the following options:
    dwc_otg.fiq_enable=0 dwc_otg.fiq_fsm_enable=0 dwc_otg.nak_holdoff=0

    Some info about this: here.

    Reboot the Raspberry Pi and if all the stars are aligned, you get the Xenomai kernel working!

    Compiling Xenomai Tools

    Xenomani includes a suite of tools for testing its realtime kernel, you can cross-compile these tools typing:

    ~/rpi-kernel$ cd xenomai-3.0.7
    ~/rpi-kernel/xenomai-3.0.7$ ./scripts/bootstrap --with-core=cobalt –enable-debug=partial
    ~/rpi-kernel/xenomai-3.0.7$ ./configure CFLAGS="-march=armv7-a -mfpu=vfp3" LDFLAGS="-mtune=cortex-a53" --build=i686-pc-linux-gnu --host=arm-linux-gnueabihf --with-core=cobalt --enable-smp CC=${CROSS_COMPILE}gcc LD=${CROSS_COMPILE}ld
    ~/rpi-kernel/xenomai-3.0.7$ make -j4 install DESTDIR=${PWD}/target

    The last line compile the tools, and it could ask you the host root password to install the tools, but don't worry they are going to be installed under the subdirectory target/.

    Again, we compress and transfer the testsuite to the Raspberry Pi using.

    ~/rpi-kernel/xenomai-3.0.7$ cd target
    ~/rpi-kernel/xenomai-3.0.7/target$ tar czf ../../xenomai-tools.tgz *
    ~/rpi-kernel/xenomai-3.0.7$ cd ~/rpi-kernel/
    ~/rpi-kernel$ scp xenomai-tools.tgz pi@<ipaddress>:/tmp

    Change again <ipaddress> to the corresponding IP of your Raspberry Pi.

    Deploying the Testsuite on Raspberry Pi

    Type the following on the Raspberry Pi:

    ~$ cd /tmp
    /tmp$ tar xzf xenomai-tools.tgz
    /tmp$ sudo cp dev/* /dev/
    /tmp$ sudo cp usr/* /usr/

    You can test the kernel latency using the testsuite:

    ~$ sudo /usr/xenomai/bin/latency 
    == Sampling period: 1000 us
    == Test mode: periodic user-mode task
    == All results in microseconds
    warming up...
    RTT|  00:00:01  (periodic user-mode task, 1000 us period, priority 99)
    RTH|----lat min|----lat avg|----lat max|-overrun|---msw|---lat best|--lat worst
    RTD|     12.239|     12.693|     19.427|       0|     0|     12.239|     19.427
    RTD|     12.291|     12.720|     16.926|       0|     0|     12.239|     19.427
    RTD|     12.290|     12.701|     17.447|       0|     0|     12.239|     19.427
    RTD|     12.238|     12.709|     14.634|       0|     0|     12.238|     19.427

    or the kernel installation using:

    ~$ sudo /usr/xenomai/bin/xeno-test

    and you get a huge report. ;)

    Latency Kernel Results

    The latency of the Xenomai patched kernel is not as good as I expected! I think, I need to check the kernel configuration again.

    Model B+ Latency Results Preempt-RT Model B+ Latency Results Xenomai
    Fig. 1a: Latency results Preempt-RT patched Kernel Fig. 1b: Latency results Xenomai patched Kernel

    The latency results were calculated using the suite RT-Tests, and the test were configured as described here.


    Comments

    Volodymyr 10.30.2018

    I used Linux Linux 4.14.71 (commit 1244bbb3e92135d247e2dddfa6fe5e3e171a9635) with patch https://xenomai.org/downloads/ipipe/v4.x/arm/ipipe-core-4.14.71-arm-4.patch

    Volodymyr 10.30.2018

    I think that at Tools the following line is cut: export CROSS_COMPILE=~/rpi-kernel/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin/arm-linux-gnueabihf-

    Thomas 11.07.2018

    Hi, very thanks to your experience. But I have a Pi 3 Model B+ and I try to build the raspberrypi/linux for it. Unfortunately, my kernel7.img was not working on RPi 3 B+. I don't find Xenomai support to Pi 3 Model B+ on the ' https://gitlab.denx.de/Xenomai/xenomai/wikis/Supported_Hardware'. Could you give me some advice? Thank you.

    Favereau Marc 11.29.2018

    Hi, I have two problems : 1) When i do "/tmp$ sudo cp dev/* /dev/" for xenomail-tools i have the message : "Can't open 'dev/rtp0' for reading: No peripherals or adress" and i can't copy 2) When i do xeno-config --cc i have the path to gcc on the dev computer and not the raspberry path, how can i change it ? I get the last linux kernel and patch : ipipe-core-4.14.71-arm-4.patch

    Thanks for four help, Best Regards.

    Victoria Tegg 04.29.2019

    This is really interesting, You’re a very skilled blogger. I have joined your feed and look forward to seeking more of your great post. Also, I’ve shared your site in my social networks!