EIVE On-Board Software

General information

Target systems:

• OBC with Linux OS
  • Xiphos Q7S
  • Based on Zynq-7020 SoC (xc7z020clg484-2)
    • Dual-core ARM Cortex-A9
    • 766 MHz
    • Artix-7 FPGA (85K pogrammable logic cells)
  • Datasheet at https://eive-cloud.irs.uni-stuttgart.de/index.php/apps/files/?dir=/EIVE_IRS/Arbeitsdaten/08_Used%20Components/Q7S&fileid=340648
  • Also a lot of information about the Q7S can be found on the Xiphos Traq Platform. Press on index to find all relevant pages.
  • Linux OS built with Yocto 2.5
  • Linux Kernel https://github.com/XiphosSystemsCorp/linux-xlnx.git . EIVE version can be found here . Pre-compiled files can be found here.
  • Q7S base project can be found here
  • Minimal base project files can be found here
• Host System
  • Generic software components which are not dependant on hardware can also be run on a host system. All host code is contained in the bsp_hosted folder
  • Tested for Linux (Ubuntu 20.04) and Windows 10
• Raspberry Pi
  • EIVE OBC can be built for Raspberry Pi as well (either directly on Raspberry Pi or by installing a cross compiler)

The steps in the primary README are related to the main OBC target Q7S. The CMake build system can be used to generate build systems as well (see helper scripts in cmake/scripts:

• Linux (Raspberry Pi): See special section below.
• Linux Host: Uses the bsp_hosted BSP folder and the CMake Unix Makefiles generator.
• Windows Host: Uses the bsp_hosted BSP folder, the CMake MinGW Makefiles generator and MSYS2.

Setting up development environment

Installing Vivado the the Xilinx development tools

It's also possible to perform debugging with a normal Eclipse installation by installing the TCF plugin and downloading the cross-compiler as specified in the section below. However, if you want to generate the *.xdi files necessary to update the firmware, you need to installed Vivado with the SDK core tools.

• Install Vivado 2018.2 and Xilinx SDK from https://www.xilinx.com/support/download/index.html/content/xilinx/en/downloadNav/vivado-design-tools/archive.html. Install the Vivado Design Suite - HLx Editions - 2018.2 Full Product Installation instead of the updates. It is recommended to use the installer.

• Install settings. In the Devices selection, it is sufficient to pick SoC → Zynq-7000:
  

• For supported OS refer to https://www.xilinx.com/support/documentation/sw_manuals/xilinx2018_2/ug973-vivado-release-notes-install-license.pdf . Installation was tested on Windows and Ubuntu 21.04.
• Add path of linux cross-compiler to permanent environment variables (.bashrc file in Linux): <XilinxInstallation>\SDK\2018.2\gnu\aarch32\nt\gcc-arm-linux-gnueabi\bin or set up path each time before debugging.

Installing on Linux - Device List Issue

When installing on Ubuntu, the installer might get stuck at the Generating installed device list step. When this happens, you can kill the installation process (might be necessara to kill a process twice) and generate this list manually with the following commands, according to this forum entry.

1. Install the following library

   sudo apt install libncurses5
   

2. sudo <installRoot>/Vivado/2018.2/bin/vivado -nolog -nojournal -mode batch -source 
   <installRoot>/.xinstall/Vivado_2018.2/scripts/xlpartinfo.tcl -tclargs 
   <installRoot>/Vivado/2018.2/data/parts/installed_devices.txt
   

For Linux, you can also download a more recent version of the Linaro 8.3.0 cross-compiler from here

Installing toolchain without Vivado

You can download the toolchains for Windows and Linux from the EIVE cloud.

If wget is available (e.g. MinGW64), you can use the following command to download the toolchain for Windows

wget https://eive-cloud.irs.uni-stuttgart.de/index.php/s/rfoaistRd67yBbH/download/gcc-arm-linux-gnueabi-win.zip

or the following command for Linux (could be useful for CI/CD)

wget https://eive-cloud.irs.uni-stuttgart.de/index.php/s/2Fp2ag6NGnbtAsK/download/gcc-arm-linux-gnueabi.tar.gz

Installing CMake and MSYS2 on Windows

1. Install MSYS2 and CMake first.

2. Open the MinGW64 console. It is recommended to set up aliases in .bashrc to navigate to the software repository quickly

3. Run the following commands in MinGW64

   pacman -Syu
   

   It is recommended to install the full base development toolchain

   pacman -S base-devel
   

   It is also possible to only install required packages

   pacman -S mingw-w64-x86_64-cmake mingw-w64-x86_64-make mingw-w64-x86_64-gcc mingw-w64-x86_64-gdb python3
   

Installing CMake on Linux

1. Run the following command

   sudo apt-get install cmake
   

Getting the Q7S system root

It is necessary to copy the Q7S system root to your local development machine for libraries like libgpio. You can find the system root for the Q7S, the Raspberry Pi and the Beagle Bone Black for download here here. Download it and unzip it somewhere in the Xilinx installation folder. You can use the following command if wget can be used or for CI/CD:

wget https://eive-cloud.irs.uni-stuttgart.de/index.php/s/agnJGYeRf6fw2ci/download/cortexa9hf-neon-xiphos-linux-gnueabi.tar.gz

Then, create a new environmental variables Q7S_SYSROOT and set it to the local system root path.

Building the software with CMake

When using Windows, run theses steps in MSYS2.

1. Clone the repository with

   git clone https://egit.irs.uni-stuttgart.de/eive/eive_obsw.git
   

2. Update all the submodules

   git submodule init
   git submodule sync
   git submodule update
   

3. Ensure that the cross-compiler is working with arm-linux-gnueabihf-gcc --version. It is recommended to run the shell script win_path_helper_xilinx_tools.sh in cmake/scripts/Q7S or to set up the PATH and the CROSS_COMPILE variable permanently in the .profile file.

4. Run the CMake configuration to create the build system in a build-Debug-Q7S folder. Add -G "MinGW Makefiles in MinGW64 on Windows.

   mkdir build-Debug-Q7S && cd build-Debug-Q7S
   cmake -DTGT_BSP="arm/q7s" -DCMAKE_BUILD_TYPE=Debug -DOS_FSFW=linux ..
   cmake --build . -j
   

   You can also use provided shell scripts to perform these commands

   cd cmake/scripts/Q7S
   ./make_debug_cfg.sh
   cd ../../..
   

   This will invoke a Python script which in turn invokes CMake with the correct arguments to configure CMake for Q7S cross-compilation.

   You can build the hosted variant of the OBSW by replacing -DOS_FSFW=linux with -DOS_FSFW=host. There are also different values for -DTGT_BSP to build for the Raspberry Pi or the Beagle Bone Black: arm/raspberrypi and arm/beagleboneblack.

5. Build the software with

   cd Debug
   cmake --build . -j
   

Setting up default Eclipse for Q7S projects - TCF agent

The TCF agent can be used to perform remote debugging on the Q7S.

1. Install the TCF agent plugin in Eclipse from the releases. Go to Help → Install New Software and use the download page, for example https://download.eclipse.org/tools/tcf/releases/1.6/1.6.2/ to search for the plugin and install it.

2. Go to Window → Perspective → Open Perspective and open the Target Explorer Perspective. Here, the Q7S should show up if the local port forwarding was set up as explained previously. Please note that you have to connect to localhost and port 1534 with port forwaring set up.

3. A launch configuration was provided, but it might be necessary to adapt it for your own needs. Alternatively:

• Create a new TCF Remote Application by pressing the cogs button at the top or going to Run → Debug Configurations → Remote Application and creating a new one there.

• Set up the correct image in the main tab (it might be necessary to send the image to the Q7S manually once) and file transfer properties

• It is also recommended to link the correct Eclipse project.

After that, comfortable remote debugging should be possible with the Debug button.

A build configuration and a shell helper script has been provided to set up the path variables and build the Q7S binary on Windows, but a launch configuration needs to be newly created because the IP address and path settings differ from machine to machine.

Building in Xilinx SDK 2018.2

1. Open Xilinx SDK 2018.2
2. Import project
   • File → Import → C/C++ → Existing Code as Makefile Project
3. Set build command. Replace <target> with either debug or release.
   • When on Linux right click project → Properties → C/C++ Build → Set build command to make <target> -j
     • -j causes the compiler to use all available cores
     • The target is used to either compile the debug or the optimized release build.
   • On windows create a make target additionally (Windows → Show View → Make Target)
     • Right click eive_obsw → New
     • Target name: all
     • Uncheck "Same as the target name"
     • Uncheck "Use builder settings"
     • As build command type: cmake --build .
     • In the Behaviour tab, you can enable build acceleration
4. Run build command by double clicking the created target or by right clicking the project folder and selecting Build Project.

TCF-Agent

1. On reboot, some steps have to be taken on the Q7S. Set static IP address and netmask

   ifconfig eth0 192.168.133.10
   ifconfig eth0 netmask 255.255.255.0
   

2. tcfagent application should run automatically but this can be checked with

   systemctl status tcfagent
   

3. If the agent is not running, check whether agent is located inside usr/bin. You can run it manually there. To perform auto-start on boot, have a look at the start-up application section.

Debugging the software via Flatsat PC

Open SSH connection to flatsat PC:

ssh eive@flatsat.eive.absatvirt.lw

or

ssh eive@2001:7c0:2018:1099:babe:0:e1fe:f1a5

or

ssh eive@192.168.199.227

If the static IP address of the Q7S has already been set, you can access it with ssh

ssh root@192.168.133.10

If this has not been done yet, you can access the serial console of the Q7S like this to set it

picocom -b 115200 /dev/ttyUSB0

If the serial port is blocked for some reason, you can kill the process using it with q7s_kill.

You can use AltGr + X to exit the picocom session.

To debug an application, first make sure a static IP address is assigned to the Q7S. Run ifconfig on the Q7S serial console.

ifconfig

Set IP address and netmask with

ifconfig eth0 192.168.133.10
ifconfig eth0 netmask 255.255.255.0

To launch application from Xilinx SDK setup port fowarding on the development machine (not on the flatsat!)

ssh -L 1534:192.168.133.10:1534 eive@2001:7c0:2018:1099:babe:0:e1fe:f1a5 -t bash

This forwards any requests to localhost:1534 to the port 1534 of the Q7S with the IP address 192.168.133.10. This needs to be done every time, so it is recommended to create an alias to do this quickly.

Note: When now setting up a debug session in the Xilinx SDK or Eclipse, the host must be set to localhost instead of the IP address of the Q7S.

Transfering files via SCP

To transfer files from the local machine to the Q7S, use port forwarding

ssh -L 1535:192.168.133.10:22 eive@2001:7c0:2018:1099:babe:0:e1fe:f1a5

An example file can be copied like this

scp -P 1535 example root@localhost:/tmp

Copying a file from Q7S to flatsat PC

scp -P 22 root@192.168.133.10:/tmp/kernel-config /tmp

From a windows machine files can be copied with putty tools (note: use IPv4 address)

pscp -scp -P 22 eive@192.168.199.227:</directory-to-example-file/>/example-file </windows-machine-path/>

Launching an application at start-up

Load the root partiton from the flash memory (there are to nor-flash memories and each flash holds two xdi images). Note: It is not possible to modify the currently loaded root partition, e.g. creating directories. To do this, the parition needs to be mounted.

1. Disable write protection of the desired root partition

   writeprotect 0 0 0 # unlocks nominal image on nor-flash 0  
   

2. Mount the root partition

   xsc_mount_copy 0 0 # Mounts the nominal image from nor-flash 0
   

   The mounted partition will be located inside the /tmp folder

3. Copy the executable to /usr/bin

4. Make sure the permissions to execute the application are set

   chmod +x application
   

5. Create systemd service in /lib/systemd/system. The following shows an example service.

   cat > example.service
   [Unit]
   Description=Example Service
   StartLimitIntervalSec=0
   
   [Service] 
   Type=simple
   Restart=always
   RestartSec=1
   User=root
   ExecStart=/usr/bin/application
   
   [Install]
   WantedBy=multi-user.target
   

6. Enable the service. This is normally done with systemctl enable. However, this is not possible when the service is created for a mounted root partition. Therefore create a symlink as follows.

   ln -s '/tmp/the-mounted-xdi-image/lib/systemd/system/example.service' '/tmp/the-mounted-xdi-image/etc/systemd/system/multi-user.target.wants/example.service'
   

7. The modified root partition is written back when the partion is locked again.

   writeprotect 0 0 1 
   

8. Now verify the application start by booting from the modified image

   xsc_boot_copy 0 0
   

9. After booting verify if the service is running

   systemctl status example
   

More detailed information about the used q7s commands can be found in the Q7S user manual.

Setting up UNIX environment for real-time functionalities

Please note that on most UNIX environments (e.g. Ubuntu), the real time functionalities used by the UNIX pthread module are restricted, which will lead to permission errors when creating these tasks and configuring real-time properites like scheduling priorities.

To solve this issues, try following steps:

1. Edit the /etc/security/limits.conf file and add following lines at the end:

<username>   hard   rtprio  99
<username>   soft   rtprio  99

The soft limit can also be set in the console with ulimit -Sr if the hard limit has been increased, but it is recommended to add it to the file as well for convenience. If adding the second line is not desired for security reasons, the soft limit needs to be set for each session. If using an IDE like eclipse in that case, the IDE needs to be started from the console after setting the soft limit higher there. After adding the two lines to the file, the computer needs to be restarted.

It is also recommended to perform the following change so that the unlockRealtime script does not need to be run anymore each time. The following steps raise the maximum allowed message queue length to a higher number permanently, which is required for some framework components. The recommended values for the new message length is 130.

2. Edit the /etc/sysctl.conf file

   sudo nano /etc/sysctl.conf
   

   Append at end:

   fs/mqueue/msg_max = <newMsgMaxLen>
   

   Apply changes with:

   sudo sysctl -p
   

   A possible solution which only persists for the current session is

   echo <newMsgMax> | sudo tee /proc/sys/fs/mqueue/msg_max
   

PCDU

Connect to serial console of P60 Dock

picocom -b 500000 /dev/ttyUSBx

General information

cmp ident

List parameter table: x values: 1,2 or 4

param list x

Table 4 lists HK parameters Changing parameters First switch to table where parameter shall be changed (here table id is 1)

p60-dock # param mem 1
p60-dock # param set out_en[0] 1
p60-dock # param get out_en[0]
GET out_en[0] = 1

Debugging the software (when workstation is directly conncected to Q7S)

1. Assign static IP address to Q7S

   • Open serial console of Q7S (Accessible via the micro-USB of the PIM, see also Q7S user manual chapter 10.3)
   • Baudrate 115200
   • Login to Q7S:
     • user: root
     • pw: root

2. Connect Q7S to workstation via ethernet

3. Make sure the netmask of the ehternet interface of the workstation matches the netmask of the Q7S

   • When IP address is set to 192.168.133.10 and the netmask is 255.255.255.0, an example IP address for the workstation is 192.168.133.2

4. Run tcf-agent on Q7S

   • Tcf-agent is not yet integrated in the rootfs of the Q7S. Therefore build tcf-agent manually

   git clone git://git.eclipse.org/gitroot/tcf/org.eclipse.tcf.agent.git
   cd org.eclipse.tcf.agent/agent
   make CC=arm-linux-gnueabihf-gcc LD=arm-linux-gnueabihf-ld MACHINE=arm NO_SSL=1 NO_UUID=1
   

   • Transfer executable agent from org.eclipse.tcf.agent/agent/obj/GNU/Linux/arm/Debug to /tmp of Q7S

   cd obj/GNU/Linux/arm/Debug
   scp agent root@192.168.133.10:/tmp
   

   • On Q7S

   cd /tmp
   chmod +x agent
   

   • Run agent

   ./agent
   

5. In Xilinx SDK 2018.2 right click on project → Debug As → Debug Configurations

6. Right click Xilinx C/C++ applicaton (System Debugger) → New →

7. Set Debug Type to Linux Application Debug and Connectin to Linux Agent

8. Click New

9. Give connection a name

10. Set Host to static IP address of Q7S. e.g. 192.168.133.10

11. Test connection (This ensures the TCF Agent is running on the Q7S)

12. Select Application tab

    • Project Name: eive_obsw
    • Local File Path: Path to eiveobsw-linux.elf (in _bin\linux\devel)
    • Remote File Path: /tmp/eive_obsw.elf

Running cppcheck on the Software

Static code analysis can be useful to find bugs. cppcheck can be used for this purpose. On Windows you can use MinGW64 to do this.

pacman -S mingw-w64-x86_64-cppcheck

On Ubuntu, install with

sudo apt-get install cppcheck

You can use the Eclipse integration or you can perform the scanning manually from the command line. CMake will be used for this.

Run the CMake build generation commands specified above but supply -DCMAKE_EXPORT_COMPILE_COMMANDS=ON to the build generation. Invoking the build command will generate a compile_commands.json file which can be used by cppcheck.

cppcheck --project=compile_commands.json --xml 2> report.xml

Finally, you can convert the generated .xml file to HTML with the following command

cppcheck-htmlreport --file=report.xml --report-dir=cppcheck --source-dir=..

Special notes on Eclipse

When using Eclipse, there are two special build variables in the project properties → C/C++ Build → Build Variables called Q7S_SYSROOT or RPI_SYSROOT. You can set the sysroot path in those variables to get any additional includes like gpiod.h in the Eclipse indexer.

Q7S Utilities and Troubleshooting

Core commands

Display currently running image:

xsc_boot_copy

Rebooting currently running image:

xsc_boot_copy -r

pa3tool Host Tool

The pa3tool is a host tool to interface with the ProASIC3 on the Q7S board. It was installed on the clean room PC but it can also be found on the Traq platform.

For more information, see Q7S datasheet.

Creating files with cat and echo

The only folder which can be written in the root filesystem is the tmp folder.

You can create a simple file with initial content with echo

echo "Hallo Welt" > /tmp/test.txt
cat /tmp/test.txt

For more useful combinations, see this link.

Using the scratch buffer of the ProASIC3

The ProASIC3 has a 1024 byte scratch buffer. The values in this scratch buffer will survive a reboot, so this buffer can be used as an alternative to the SD cards to exchange information between images or to store mission critical information.

You can use xsc_scratch --help for more information.

Write to scratch buffer:

xsc_scratch write TEST "1"

Read from scratch buffer:

xsc_scratch read TEST

Read all keys:

xsc_scratch read

Get fill count:

xsc_scratch read | wc -c

Using system when debugging

Please note that when using a system call in C++/C code and debugging, a new thread will be spawned which will appear on the left in Eclipse or Xilinx SDK as a sh program. The debugger might attach to this child process automatically, depending on debugger configuration, and the process needs to be selected and continued/started manually. You can enable or disable this behaviour by selecting or deselecting the Attach Process Children option in the Remote Application Configuration for the TCF plugin like shown in the following picture

Libgpiod

Detect all gpio device files:

gpiodetect

Get info about a specific gpio group:

gpioinfo <name of gpio group>

The following sets the gpio 18 from gpio group gpiochip7 to high level.

gpioset gpiochip7 18=1

Setting the gpio to low.

gpioset gpiochip7 18=0

Show options for setting gpios.

gpioset -h

To get the state of a gpio:

gpioget <gpiogroup> <offset>

Example to get state: gpioget gpiochip7 14

Both the MIOs and EMIOs can be accessed via the zynq_gpio instance which comprises 118 pins (54 MIOs and 64 EMIOs).

Xilinx UARTLIE

Get info about ttyUL* devices

cat /proc/tty/driver

I2C

Getting information about I2C device

ls /sys/class/i2c-dev/i2c-0/device/device/driver

This shows the memory mapping of /dev/i2c-0

CAN

ip link set can0 down
ip link set can0 type can loopback off
ip link set can0 up type can bitrate 1000000

Following command sends 8 bytes to device with id 99 (for petalinux)

cansend can0 -i99 99 88 77 11 33 11 22 99

For Q7S use this:

cansend can0 5A1#11.22.33.44.55.66.77.88

Turn loopback mode on:

ip link set can0 type can bitrate 1000000 loopback on 

Reading data from CAN:

candump can0

Preparation of a fresh rootfs and SD card

This section summarizes important changes between a fresh rootfs and the current EIVE implementation

rootfs

• Mount point /mnt/sd0 created for SD card 0. Created with mkdir
• Mount point /mnt/sd1 created for SD card 1. Created with mkdir
• Folder scripts in /home/root folder.
• scripts folder currently contains update_main_components.sh script

SD Cards

• Folder bin for binaries, for example the OBSW
• Folder misc for miscellaneous files. Contains ls for directory listings
• Folder tc for telecommands
• Folder tm for telemetry
• Folder xdi for XDI components (e.g. for firmware or device tree updates)

Running the EIVE OBSW on a Raspberry Pi

Special section for running the EIVE OBSW on the Raspberry Pi. The Raspberry Pi build uses the bsp_rpi BSP folder, and a very similar cross-compiler.

For running the software on a Raspberry Pi, it is recommended to follow the steps specified in the fsfw example and using the TCF agent to have a similar set-up process also required for the Q7S. You should run the following command first on your Raspberry Pi

sudo apt-get install gpiod libgpiod-dev

to install the required GPIO libraries before cloning the system root folder.

Flight Software Framework (FSFW)

An EIVE fork of the FSFW is submodules into this repository. To add the master upstream branch and merge changes and updates from it into the fork, run the following command in the fsfw folder first:

git remote add upstream https://egit.irs.uni-stuttgart.de/fsfw/fsfw.git
git remote update --prune

After that, an update can be merged by running

git merge upstream/master

Alternatively, changes from other upstreams (forks) and branches can be merged like that in the same way.