fsfw-example-linux-mcu/doc/README-rpi.md

<img align="center" src="./images/rpi/RPi-Logo-Landscape-Reg-PRINT.png" width="30%">

<sub><sup>Image taken from [Raspberry Pi website](https://www.raspberrypi.org/trademark-rules/).
Raspberry Pi is a trademark of the Raspberry Pi Foundation</sup></sub>

Getting started on the Raspberry Pi
======
The FSFW can be run on a Raspberry Pi with the Linux OSAL, using
an ARM linux (cross) compiler. Instructions will be provided on how 
to do this.

# General Information

The following instructions will show how to build the example on the Raspberry Pi directly.
It will also show how to cross-compile on a host machine and mirror the Raspberry Pi sysroot folder
on the host machine so that the same libraries and headers used on the RPi are used
for the cross-compilation process.

Some Eclipse project files were provided as well to help with setting up the indexer in Eclipse
more quickly.

# Prerequisites for direct compilation and cross-compiling

1. SSH connection to the Raspberry Pi working 
2. Raspberry Pi linux environment set up properly
3. `CMake` installed

# Setting up general prerequisites for Linux systems

1. Install `CMake` and `rsync`

    ```sh
    sudo apt-get install cmake rsync
    ```

2. Configure the Raspberry Pi Linux environment. The last section of the 
   [Linux README](README-linux.md#top) specifies how to set up a UNIX environment for the FSFW and 
   is also applicable to the Raspberry Pi. SSH into the Raspberry Pi and 
   follow the instructions in that section.
 
3. Install the `gpiod` library

   ```sh
   sudo apt-get install gpiod libgpiod-dev
   ```

# Getting started on the Raspberry Pi

Make sure to follow the steps above. Now you should be able to build the software on 
the Raspberry Pi. A ssh connection to the Raspberry Pi is assumed here

You can build the software with the following commands

```sh 
mkdir build-Debug-RPi
cd build-Debug-RPi
cmake -DOS_FSFW=linux -DTGT_BSP=arm/raspberrypi -DLINUX_CROSS_COMPILE=OFF -DCMAKE_BUILD_TYPE=Debug ..
cmake --build . -j
```

# Prerequisites for cross-compiling

These prerequisites are valid for Linux as well as Windows hosts.

1. ARM Linux cross compiler installed
2. Raspberry Pi sysroot folder mirrored on the host machine, using `rsync` or `scp`
3. gdb-multiarch installed on host for remote debugging or `tcf-agent` running on Raspberry Pi

# Cross-Compiling on a Linux Host

Steps tested for Ubuntu 20.04. Adapt accordingly for used Linux distribution.
The following steps are based on this 
[stackoverflow post](https://stackoverflow.com/questions/19162072/how-to-install-the-raspberry-pi-cross-compiler-on-my-linux-host-machine). 
For the steps show here, we are also going to assume that a new Raspbian image 
based on Debian buster is used. If this is not the case, it is recommended to 
follow the steps in the stackoverflow post above and to make sure that the 
toolchain binaries are added to the path accordingly.

## Setting up prerequisites for cross-compiling

You can skip step 1 if you already have a cross-compile toolchain or you can
cross-compile a toolchain yourself by following the steps in the 
[crosstool-ng chapter](#ctng).

1. Install the pre-built ARM cross-compile with the following command

   ```sh
   wget https://github.com/Pro/raspi-toolchain/releases/latest/download/raspi-toolchain.tar.gz
   ```

   Then extract to the opt folder:
   
   ```sh
   sudo tar xfz raspi-toolchain.tar.gz --strip-components=1 -C /opt
   ```
   
   Please note that this version of the toolchain might become obsolete in the future.
   If another toolchain installation is used, it is still recommended to unpack the toolchain in the 
   `/opt/cross-pi-gcc` folder so that the Eclipse configuration and helper 
   scripts work without adaptions. Add the folder to the system path. On Linux,
   this can generally be done with the following command

   ```sh
   export PATH=$PATH:"/opt/cross-pi-gcc/bin"
   ``` 

   You can add this line to the `.bashrc` or `.profile` file in the `$HOME` directory 
   to add environmental variables permanently. More experienced users can
   perform this step is a shell script which is `source`d to keep the environment clean.

   Test the toolchain with the following command

   ```sh
   arm-linux-gnueabihf-gcc --version
   ``` 

2. Set up a sysroot folder on the local host machine. Make sure the SSH connection to
   the Raspberry Pi is working without issues. Then perform the following steps

   ```sh
   cd ~
   mkdir raspberrypi
   cd raspberrypi
   mkdir rootfs
   cd rootfs 
   pwd
   ```

   The result of the `pwd` command will be used later to sync the root file
   system of the Raspberry Pi to the host machine.
   With a Raspberry Pi 4, you can replace `<ip-address>` with `raspberrypi.local` and 
   when using the default rootfs path, you can replace `<rootfs-path>` with
   `$HOME/raspberrypi/rootfs`.

   ```sh
   rsync -avHAXR --numeric-ids --info=progress2 <username>@<ip-address>:/{lib,usr,opt/vc/lib} <rootfs-path>
   ```

   On Linux, it is recommended to repair some symlinks which can be problematic:
   Navigate to the folder containing the symlinks first:

   ```sh
   cd <rootfs_path>/usr/lib/arm-linux-gnueabihf
   ```

   You can now use

   ```sh
   readlink libpthread.so
   ```

   which will show an absolute location of a shared library the symlinks points to. This location
   needs to be converted into a relative path.

   Run the following command to create a relative symlinks instead of an absolute ones. The pointed
   to location might change to check it with `readlink` first before removing the symlinks:

   ```sh
   rm libpthread.so
   rm librt.so
   ln -s ../../../lib/arm-linux-gnueabihf/libpthread.so.0 libpthread.so
   ln -s ../../../lib/arm-linux-gnueabihf/librt.so.1 librt.so
   ```

   For more information on issues which can occur when cloning the root filesystem,
   see the [troubleshooting](#troubleshooting) section.

3. It is recommended to install `gdb-multiarch`. This tool will allow remote debugging
   on the host computer. This step is not required if the `tcf-agent` is used.

   ```sh
   sudo apt-get install gdb-multiarch
   ```

4. Perform the steps [in the cross-compile section](#cross-test) to build the 
   software for the Raspberry Pi and test it.

# Cross-Compiling on a Windows Host

## Additional Prerequites

1. [MSYS2](https://www.msys2.org/) installed. All command line steps shown here
   were performed in the MSYS2 MinGW64 shell (not the default MSYS2, use MinGW64!).
   Replace `<UserName>` with respectively. It is recommended to set up
   aliases in the `.bashrc` file to allow quick navigation to the `fsfw_example`
   repository and to run `git config --global core.autocrlf true` for git in
   MinGW64.

## Setting up prerequisites for Windows

1. Install CMake and rsync in MinGW64 after installing MSYS2

   ```
   pacman -S mingw-w64-x86_64-cmake rsync
   ```

2. Configure the Raspberry Pi linux environment. The last section of the 
   [Linux REAMDE](README-linux.md#top) specifies how to set up a UNIX environment 
   for the FSFW and isalso applicable to the Raspberry Pi. SSH into the 
   Raspberry Pi and follow the instructions in that section.

3. Install the correct [ARM Linux cross-compile toolchain provided by SysProgs](https://gnutoolchains.com/raspberry/).
   You can find out the distribution release of your Raspberry Pi by running `cat /etc/rpi-issue`.

   Test the toolchain by running:

   ```sh
   arm-linux-gnueabihf-gcc --version
   ``` 

4. Set up a sysroot folder on the local host machine. Make sure the SSH connection to
   the Raspberry Pi is working without issues. Then perform the following steps

   ```sh
   cd /c/Users/<UserName>
   mkdir raspberrypi
   cd raspberrypi
   mkdir rootfs
   cd rootfs
   pwd
   ```

   Store the result of `pwd`, it is going to be used by `rsync` later.

   Now use rsync to clone the Rapsberry Pi sysroot to the local host machine.
   With a Raspberry Pi 4, you can replace `<ip-address>` with `raspberrypi.local`.
   Use the rootfs location stored from the previous steps as `<rootfs-path>`.

   ```sh
   rsync -vR --progress -rl --delete-after --safe-links pi@<ip-address>:/{lib,usr,opt/vc/lib} <rootfs-path>
   ```

   Please note that `rsync` sometimes does not copy shared libraries or symlinks properly,
   which might result in errors when cross-compiling and cross-linking. It is recommended to run
   the following commands in addition to the `rsync` command on Windows:

   ```sh
   scp <user_name>@<ip-address>:/lib/arm-linux-gnueabihf/{libc.so.6,ld-linux-armhf.so.3,libm.so.6} <rootfs_path>/lib/arm-linux-gnueabihf
   scp <user_name>@<ip-address>:/usr/lib/arm-linux-gnueabihf/{libpthread.so,libc.so,librt.so} <rootfs_path>/usr/lib/arm-linux-gnueabihf
   ```

   For more information on issues which can occur when cloning the root filesystem,
   see the [troubleshooting](#troubleshooting) section.

5. It is recommended to install `gdb-multiarch`. This tool will allow remote debugging on the host 
   computer. Replace `x86_64` with the correct processor architecture for other architectures.
   This is not required if the `tcf-agent` is used.

   ```sh
   pacman -S mingw-w64-x86_64-gdb-multiarch
   ```

6. Perform the steps [in the following chapter](#cross-test) to build the
   software for the Raspberry Pi and test it.

# <a id="cross-test"></a> Testing the cross-compilation

It is recommended to set the following environmental variables for the CMake build:
   - `CROSS_COMPILE`: Explicitely specify the name of the cross compiler 
   - `RASPBERRY_VERSION`: Explicitely specify the version of the Raspberry Pi
   - `LINUX_ROOTFS`: Explicitely set the path to the local RPi rootfs

For example with the following commands

```sh
export CROSS_COMPILE="arm-linux-gnueabihf"
export RASPBERRY_VERSION="4"
export LINUX_ROOTFS="<pathToRootFS>"
```

It is recommended to test whether the environmental variables were set correctly,
for example by running

```sh
echo $RASPBIAN_ROOTFS
```

These variables can either be set every time before a debugging session to
keep the environment clean (should be done before starting Eclipse)
or permanently by adding the `export` commands to system files.

A helper script has been provided in `cmake/scripts/RPi` to perform
setting up the environment. The scripts need to be `source`d instead of 
being run like regular shell scripts. 

You can also set up the environmental variables permanently by adding the 
export commands to the `.profile` or `.bashrc` file in the `$HOME` folder.
On Windows, MinGW64 was used to set up the build system, so you can use the
MinGW64 `.bashrc` file to do this. If you are using Eclipse to build 
the software, Eclipse will have the system variables from Windows,
so it is recommended to either permanently set the three environmental
variables in the Windows system environmental variables or add them in
Eclipse. See the [Eclipse README](README-eclipse.md#top) for more information.

Now we can test whether everything was set up properly by compiling the example
and running it on the Raspberry Pi via command line. 
Navigate into the `fsfw_example` folder first. 

1. Build the software locally to test the cross-compilation process. 
   We are going to create a Debug build directory first.

   ```sh 
   mkdir build-Debug-RPi
   cd build-Debug-RPi
   ```

2. Configure the build system. On Linux, run the following command:

   ```sh
   cmake -G "Unix Makefiles" -DOS_FSFW=linux -DTGT_BSP=arm/raspberrypi -DLINUX_CROSS_COMPILE=ON -DCMAKE_BUILD_TYPE=Debug ..
   ```

   On Windows, replace `-G "Unix Makefiles"` with `-G "MinGW Makefiles"`.

   Alternatively, you can use the helper shell scripts located inside `cmake/scripts/RPi/crosscompile`
   or the Python helper script `cmake_build_config.py` inside the `cmake/scripts` folder.
   The `RPi` folder also contains template shell files which can be `source`d
   to quickly set up the environmental variables if you want to keep the system path clean.

3. Run the binary to test it

   ```sh
   scp fsfw_example pi@raspberrypi.local:/home/pi/fsfw_example
   ssh pi@raspberrypi.local
   ./fsfw_example
   ```

## Setting up Eclipse for a Raspberry Pi remote target

It is recommended to use the provided Eclipse project files and
launch configurations to have a starting point. See the specific section in
the [Eclipse README](README-eclipse.md#top) for information how to do this.

### Windows

There are some additional steps necessary on Windows: The cross-compiler by 
default is configured to look for the cross-compiler in `/opt/cross-pi-gcc/bin`. 
The toolchain path needs to be corrected, for example like shown in the following image:

<img align="center" src="./images/eclipse/eclipse-cross-compile-win.png" width="50%">

# Setting up the TCF agent on the Raspberry Pi

It is recommended to set up a [TCF agent](https://wiki.eclipse.org/TCF) for comfortable
Eclipse remote debugging. The following steps show how to setup the TCF agent 
on the Raspberry Pi and add it to the auto-startup applications. The steps are taken
from [this guide](https://wiki.eclipse.org/TCF/Raspberry_Pi)

1. Install required packages on the RPi

   ```sh
   sudo apt-get install git uuid uuid-dev libssl-dev
   ```

2. Clone the repository and perform some preparation steps
   ```sh
   git clone git://git.eclipse.org/gitroot/tcf/org.eclipse.tcf.agent.git
   cd org.eclipse.tcf.agent.git/agent
   ```

3. Build the TCF agent
   ```sh
   make
   ```

   and then test it by running

   ```sh
   obj/GNU/Linux/arm/Debug/agent –S
   ```

4. Finally instal lthe agent for auto-start with the following steps and set it up for auto-start.
   The last step did not work on a Rapsberry Pi 4, but apparentely was not necessary.

   ```sh
   cd org.eclipse.tcf.agent/agent
   make install
   sudo make install INSTALLROOT=
   sudo update-rc.d tcf-agent defaults
   sudo update-rc.d tcf-agent enable 2
   ```

The [Eclipse README](README-eclipse.md#top) specifies how to perform remote 
debugging using the TCF agent.

# <a id="troubleshooting"></a> Troubleshooting

## Cloning the root filesystem

There might be some issues with the pthread symbolic links.
Navigate to the folder containing the symlinks

```sh
cd <rootfs_path>/usr/lib/arm-linux-gnueabihf
```

Type `more libpthread`, press  `TAB` and check whether the symbolic 
link `libpthread.so` is shown. If it is not, we are going to set it up
manually to avoid issues when linking against `pthread` later. 
Now you can find out where `libpthread.so` points with `readlink libpthread.so`.
This information is used to convert the absolute symlink to relative ones, for example with:

Run the following command to copy the symlink `libpthread.so.0` if it does not exist yet:

```sh
scp <user_name>@<ip-address>:/usr/lib/arm-linux-gnueabihf/libpthread.so .
```

Alternatively, you can correct the symlinks to use relative paths, for example with:

```sh
ln -s ../../../lib/arm-linux-gnueabihf/libpthread.so.0 libpthread.so
ln -s ../../../lib/arm-linux-gnueabihf/librt.so.1 librt.so
```

Please note that there might also be issues with some symlinks or libraries not being copied 
properly, especially on Windows. This has occured with files like `libc.so.6`.
If there are linker issues at a later stage, you can try to copy the symlinks manually from the
Linux board to the sysroot with `scp`.

For example, you can copy `libc.so.6` from the Linux board to the sysroot with 
the following command
 
If there are issues with the cross-compilation process, manually copying the following
symlinks can help:

```sh
scp <user_name>@<ip-address>:/usr/lib/arm-linux-gnueabihf/libc.so <rootfs_path>/usr/lib/arm-linux-gnueabihf
scp <user_name>@<ip-address>:/usr/lib/arm-linux-gnueabihf/libc.a <rootfs_path>/usr/lib/arm-linux-gnueabihf

scp <user_name>@<ip-address>:/usr/lib/arm-linux-gnueabihf/librt.a <rootfs_path>/usr/lib/arm-linux-gnueabihf
scp <user_name>@<ip-address>:/usr/lib/arm-linux-gnueabihf/librt.so <rootfs_path>/usr/lib/arm-linux-gnueabihf

scp <user_name>@<ip-address>:/lib/arm-linux-gnueabihf/librt.so.1 <rootfs_path>/lib/arm-linux-gnueabihf
scp <user_name>@<ip-address>:/lib/arm-linux-gnueabihf/libpthread.so.0 <rootfs_path>/lib/arm-linux-gnueabihf
scp <user_name>@<ip-address>:/lib/arm-linux-gnueabihf/ld-linux-armhf.so.3 <rootfs_path>/lib/arm-linux-gnueabihf
scp <user_name>@<ip-address>:/lib/arm-linux-gnueabihf/libc.so.6 <rootfs_path>/lib/arm-linux-gnueabihf
```

If any custom libraries are used which rely on symlinks, it might be necessary to copy them
or create them manually as well.

# <a id="ctng"></a> Building a cross-compile toolchain with `crosstool-ng`

If you want to cross-compile the toolchain for the Raspberry Pi, you can do so
with the `crosstool-ng` tool.

Alternatively, you can also download a cross-compile toolchain built with
crosstool-ng for the Raspberry Pi 3 and the Raspbery Pi 4 
[from here](https://www.dropbox.com/sh/zjaex4wlv5kcm6q/AAABBFfmZSRZ7GE7ok-7vTE6a?dl=0)
inside the `x-tools` folder.

## Ubuntu

1. Install prerequisites to build toolchains

   ```sh
   sudo apt-get install automake bison chrpath flex g++ git gperf gawk help2man libexpat1-dev
   libncurses5-dev libsdl1.2-dev libtool libtool-bin libtool-doc python2.7-dev texinfo
   ```

2. Install `crosstool-ng`. You can checkout a concrete version in the git repository, we will
   simply build the master branch here:

   ```sh
   git clone https://github.com/crosstool-ng/crosstool-ng.git
   cd crosstool-ng/
   ```

3. Install `crosstool-ng`. 
   ```sh
   ./bootstrap
   ./configure --enable-local
   make
   sudo make install
   ```

   You can also install `ct-ng` locally by replacing `--enable-local` with `--prefix=<installPath>`
   to the `configure`  command. You don't need `sudo` for the `make install` command if you do this.

4. You can get a list of architectures by running 

   ```sh
   ./ct-ng list-samples > samples.txt
   cat samples.txt
   ```

   `ct-ng` includes pre-configurations for the Raspberry Pis. For example you can display the
   settings for the Raspberry Pi 3 with the command

   ```sh
   ./ct-ng show-armv8-rpi3-linux-gnueabihf
   ```

   You can add the configuration with

   ```sh
   ./ct-ng armv8-rpi3-linux-gnueabihf
   ```

   You can now customize the build with

   ```sh
   ./ct-ng menuconfig
   ```

   It is recommended to go to `Paths and misc options` and disable `Paths and misc options`.
   Remember to save the configuration.

5. The correct configuration files for the Raspberry Pi 3 and Raspberry Pi 4 are provided here
   are were set up according to [these instructions](https://medium.com/@stonepreston/how-to-cross-compile-a-cmake-c-application-for-the-raspberry-pi-4-on-ubuntu-20-04-bac6735d36df).
   You can download those files from [here](https://www.dropbox.com/sh/ok554i453phuqec/AACcv-fxpIXi3Nh1z8UWu4Sea?dl=0).

   Put the `patches` folder, the `.config` folder and the `env.sh` in the same directory where
   you run `ct-ng`. After that, you can check the configuration with `./ct-ng menuconfig`.

   Set up the multiarch environment for debian by running

   ```sh
   . env.sh
   ```

6. Finally, after finishing the configuration you can build the toolchain with

   ```sh
   ./ct-ng build
   ```

   This takes 20-30 minutes. You can find the toolchain in the `~/x-tools` folder after building
   has finished

7. You can check whether the toolchain was configured correctly for multiarch by running the
   following command

   ```sh
   cd ~/x-tools/armv8-rpi3-linux-gnueabihf/armv8-rpi3-linux-gnueabihf/bin
   ./ld --verbose | grep -i "search"
   ```

   The output should include

   ```sh
   SEARCH_DIR("=/lib/arm-linux-gnueabihf"); SEARCH_DIR("=/usr/lib/arm-linux-gnueabihf");
   ```