fsfw/fsfw
fsfw
/
fsfw
10
3
Fork 6
Code Issues 31 Pull Requests 4 Releases 10 Wiki Activity

WIP: Refactor logging system with {fmt} #620

Draft
muellerr wants to merge 31 commits from mueller/refactor-logging-with-fmt into development
pull from: mueller/refactor-logging-with-fmt
merge into: fsfw:development
Conversation 1 Commits 31 Files Changed 158 +1333 -2414
muellerr commented 2022-05-09 11:26:07 +02:00
Owner
Copy Link

Related issue: #609

  • This solves the issue of having to support two different logging facility by replacing it with a better one
  • Some tests are still necessary, considering parameters like code size
  • There is no clean way around exceptions anymore. They need to be handled in the two central logger functions. I don't have a large concern about exceptions anyway (e.g. they allow using the .at() API). Some of them are really useful and they were never explicitely disabled anyway.
  • More powerful loggers, able to display source file information including the line number
  • Adapted to avoid dynamic memory allocation: The two central logger functions write their output into a shared buffer protected with a mutex. This should also prevent garbled printout and avoids most if not all of the dynamic memory allocation
  • Having two central logger functions allow easily disabling the loggers at one point in the code instead of littering everything with preprocessor defines
  • Having those central adaptions points also allows easily repiping the output, for example into a file. {fmt} also supports writing to a file
  • New concept of logging levels. This allows in principle to set a central log level to disable FSFW output
  • This completely replaces the two old APIs
  • C++20 implements most of fmt . This makes the feature also future-proof

TODO

  • Better migration guide
  • Finish tests related to code size
  • Test this on smaller embedded systems like the STM32H7 oder the SOURCE OBC
  • Fix the tests
  • Update all examples
  • Restore array printers. {fmt} probably has helper functions for this
  • EventManager debug printout fix

Example code

Some test code in the hosted example, which displays
different ways to use the fsfw loggers

Migration information

Code Size Benchmarks

Tests done on Hosted Example Repo tag 466b088dcea7 and compiler gcc (Ubuntu 11.2.0-19ubuntu1) 11.2.0. There will be some more tests. Speed benchmarks should not be necessary. Those were covered by the author himself.

Using {fmt}, linked statically

Debug builds

LTO Enabled

[246/246] Linking CXX executable fsfw-example-hosted
   text	   data	    bss	    dec	    hex	filename
1415136	  55392	   1512	1472040	 167628	fsfw-example-hosted

LTO Disabled

[244/244] Linking CXX executable fsfw-example-hosted
   text	   data	    bss	    dec	    hex	filename
1204660	  47624	   1512	1253796	 1321a4	fsfw-example-hosted

Release builds

LTO Enabled

[246/246] Linking CXX executable fsfw-example-hosted
   text	   data	    bss	    dec	    hex	filename
 717577	  41704	   1544	 760825	  b9bf9	fsfw-example-hosted

LTO Disabled

[26/26] Linking CXX executable fsfw-example-hosted
   text	   data	    bss	    dec	    hex	filename
 821826	  44512	   1512	 867850	  d3e0a	fsfw-example-hosted

Size builds

LTO Enabled

   text	   data	    bss	    dec	    hex	filename
 477212	  43104	   1480	 521796	  7f644	fsfw-example-hosted

LTO Disabled

[244/244] Linking CXX executable fsfw-example-hosted
   text	   data	    bss	    dec	    hex	filename
 613152	  45456	   1512	 660120	  a1298	fsfw-example-hosted

Using {fmt}, shared library

Debug builds

LTO Enabled

[246/246] Linking CXX executable fsfw-example-hosted
   text	   data	    bss	    dec	    hex	filename
1367682	  55080	   1480	1424242	 15bb72	fsfw-example-hosted

LTO Disabled

[246/246] Linking CXX executable fsfw-example-hosted
   text	   data	    bss	    dec	    hex	filename
1148373	  47152	   1448	1196973	 1243ad	fsfw-example-hosted

Release builds

LTO Enabled

[246/246] Linking CXX executable fsfw-example-hosted
   text	   data	    bss	    dec	    hex	filename
 667855	  41368	   1512	 710735	  ad84f	fsfw-example-hosted

LTO Disabled

[246/246] Linking CXX executable fsfw-example-hosted
   text	   data	    bss	    dec	    hex	filename
 791737	  44096	   1448	 837281	  cc6a1	fsfw-example-hosted

Size builds

LTO Enabled

[246/246] Linking CXX executable fsfw-example-hosted
   text	   data	    bss	    dec	    hex	filename
 427299	  42752	   1480	 471531	  731eb	fsfw-example-hosted

LTO Disabled

[246/246] Linking CXX executable fsfw-example-hosted
   text	   data	    bss	    dec	    hex	filename
 564905	  45016	   1448	 611369	  95429	fsfw-example-hosted

Using iostream

tag 21b3e46a72

Debug builds

LTO Enabled

[246/246] Linking CXX executable fsfw-example-hosted
   text	   data	    bss	    dec	    hex	filename
1037821	  61704	   3424	1102949	 10d465	fsfw-example-hosted

LTO Disabled

[246/246] Linking CXX executable fsfw-example-hosted
   text	   data	    bss	    dec	    hex	filename
 867602	  48160	   3776	 919538	  e07f2	fsfw-example-hosted

Release builds

LTO Enabled

[246/246] Linking CXX executable fsfw-example-hosted
   text	   data	    bss	    dec	    hex	filename
 534420	  41496	   3336	 579252	  8d6b4	fsfw-example-hosted

LTO Disabled

[246/246] Linking CXX executable fsfw-example-hosted
   text	   data	    bss	    dec	    hex	filename
 622275	  45024	   3776	 671075	  a3d63	fsfw-example-hosted

Size builds

LTO Enabled

[246/246] Linking CXX executable fsfw-example-hosted
   text	   data	    bss	    dec	    hex	filename
 352480	  42968	   3288	 398736	  61590	fsfw-example-hosted

LTO Disabled

[246/246] Linking CXX executable fsfw-example-hosted
   text	   data	    bss	    dec	    hex	filename
 424601	  46128	   3776	 474505	  73d89	fsfw-example-hosted
Related issue: https://egit.irs.uni-stuttgart.de/fsfw/fsfw/issues/609 - This solves the issue of having to support two different logging facility by replacing it with a better one - Some tests are still necessary, considering parameters like code size - There is no clean way around exceptions anymore. They need to be handled in the two central logger functions. I don't have a large concern about exceptions anyway (e.g. they allow using the `.at()` API). Some of them are really useful and they were never explicitely disabled anyway. - More powerful loggers, able to display source file information including the line number - Adapted to avoid dynamic memory allocation: The two central logger functions write their output into a shared buffer protected with a mutex. This should also prevent garbled printout and avoids most if not all of the dynamic memory allocation - Having two central logger functions allow easily disabling the loggers at one point in the code instead of littering everything with preprocessor defines - Having those central adaptions points also allows easily repiping the output, for example into a file. `{fmt}` also supports writing to a file - New concept of logging levels. This allows in principle to set a central log level to disable FSFW output - This completely replaces the two old APIs - C++20 implements most of `fmt` . This makes the feature also future-proof # TODO - Better migration guide - Finish tests related to code size - Test this on smaller embedded systems like the STM32H7 oder the SOURCE OBC - Fix the tests - Update all examples - Restore array printers. `{fmt}` probably has helper functions for this - `EventManager` debug printout fix # Example code Some test code in the [hosted example](https://egit.irs.uni-stuttgart.de/fsfw/fsfw-example-common/src/branch/mueller/fmt-log/example/test/testFmt.cpp), which displays different ways to use the `fsfw` loggers # Migration information # Code Size Benchmarks Tests done on [Hosted Example Repo](https://egit.irs.uni-stuttgart.de/fsfw/fsfw-example-hosted) tag `466b088dcea7` and compiler `gcc (Ubuntu 11.2.0-19ubuntu1) 11.2.0`. There will be some more tests. Speed benchmarks should not be necessary. Those were covered by the author himself. ## Using {fmt}, linked statically ### Debug builds #### LTO Enabled ```sh [246/246] Linking CXX executable fsfw-example-hosted text data bss dec hex filename 1415136 55392 1512 1472040 167628 fsfw-example-hosted ``` #### LTO Disabled ```sh [244/244] Linking CXX executable fsfw-example-hosted text data bss dec hex filename 1204660 47624 1512 1253796 1321a4 fsfw-example-hosted ``` ### Release builds #### LTO Enabled ```sh [246/246] Linking CXX executable fsfw-example-hosted text data bss dec hex filename 717577 41704 1544 760825 b9bf9 fsfw-example-hosted ``` #### LTO Disabled ```sh [26/26] Linking CXX executable fsfw-example-hosted text data bss dec hex filename 821826 44512 1512 867850 d3e0a fsfw-example-hosted ``` ### Size builds #### LTO Enabled ```sh text data bss dec hex filename 477212 43104 1480 521796 7f644 fsfw-example-hosted ``` #### LTO Disabled ```sh [244/244] Linking CXX executable fsfw-example-hosted text data bss dec hex filename 613152 45456 1512 660120 a1298 fsfw-example-hosted ``` ## Using {fmt}, shared library ### Debug builds #### LTO Enabled ```sh [246/246] Linking CXX executable fsfw-example-hosted text data bss dec hex filename 1367682 55080 1480 1424242 15bb72 fsfw-example-hosted ``` #### LTO Disabled ```sh [246/246] Linking CXX executable fsfw-example-hosted text data bss dec hex filename 1148373 47152 1448 1196973 1243ad fsfw-example-hosted ``` ### Release builds #### LTO Enabled ```sh [246/246] Linking CXX executable fsfw-example-hosted text data bss dec hex filename 667855 41368 1512 710735 ad84f fsfw-example-hosted ``` #### LTO Disabled ```sh [246/246] Linking CXX executable fsfw-example-hosted text data bss dec hex filename 791737 44096 1448 837281 cc6a1 fsfw-example-hosted ``` ### Size builds #### LTO Enabled ```sh [246/246] Linking CXX executable fsfw-example-hosted text data bss dec hex filename 427299 42752 1480 471531 731eb fsfw-example-hosted ``` #### LTO Disabled ```sh [246/246] Linking CXX executable fsfw-example-hosted text data bss dec hex filename 564905 45016 1448 611369 95429 fsfw-example-hosted ``` ## Using iostream tag `21b3e46a72` ### Debug builds #### LTO Enabled ```sh [246/246] Linking CXX executable fsfw-example-hosted text data bss dec hex filename 1037821 61704 3424 1102949 10d465 fsfw-example-hosted ``` #### LTO Disabled ```sh [246/246] Linking CXX executable fsfw-example-hosted text data bss dec hex filename 867602 48160 3776 919538 e07f2 fsfw-example-hosted ``` ### Release builds #### LTO Enabled ```sh [246/246] Linking CXX executable fsfw-example-hosted text data bss dec hex filename 534420 41496 3336 579252 8d6b4 fsfw-example-hosted ``` #### LTO Disabled ```sh [246/246] Linking CXX executable fsfw-example-hosted text data bss dec hex filename 622275 45024 3776 671075 a3d63 fsfw-example-hosted ``` ### Size builds #### LTO Enabled ```sh [246/246] Linking CXX executable fsfw-example-hosted text data bss dec hex filename 352480 42968 3288 398736 61590 fsfw-example-hosted ``` #### LTO Disabled ```sh [246/246] Linking CXX executable fsfw-example-hosted text data bss dec hex filename 424601 46128 3776 474505 73d89 fsfw-example-hosted ```
muellerr added 8 commits 2022-05-09 11:26:09 +02:00
fsfw/fsfw/pipeline/head There was a failure building this commit Details
77055a1579
first working version with fmt lib
fsfw/fsfw/pipeline/head There was a failure building this commit Details
fsfw/fsfw/pipeline/pr-development There was a failure building this commit Details
8d85da66f2
remove double added source files
muellerr changed title from Refactor logging system with {fmt} to WIP: Refactor logging system with {fmt} 2022-05-09 11:29:01 +02:00
muellerr added this to the v6.0.0 milestone 2022-05-09 11:29:06 +02:00
muellerr requested review from mohr 2022-05-09 11:39:04 +02:00
muellerr requested review from gaisser 2022-05-09 11:39:04 +02:00
muellerr added 1 commit 2022-05-10 10:11:01 +02:00
fsfw/fsfw/pipeline/pr-development There was a failure building this commit Details
16bbc0f597
Merge remote-tracking branch 'origin/development' into mueller/refactor-logging-with-fmt
muellerr added 11 commits 2022-05-10 10:15:10 +02:00
fsfw/fsfw/pipeline/head Build started... Details
fsfw/fsfw/pipeline/pr-development This commit looks good Details
a943e4eebb
enable LTO where applicable
fsfw/fsfw/pipeline/pr-development This commit looks good Details
a4bd5a2aaa
update changelog
fsfw/fsfw/pipeline/pr-development There was a failure building this commit Details
637512ad77
changelog update
fsfw/fsfw/pipeline/pr-development There was a failure building this commit Details
8a40878eb5 Merge remote-tracking branch 'origin/development' into mueller/add-lto-support
fsfw/fsfw/pipeline/pr-development There was a failure building this commit Details
75132c1e39 Merge branch 'development' into mueller/add-lto-support
fsfw/fsfw/pipeline/pr-development This commit looks good Details
dd9e28fca1 Merge branch 'development' into mueller/add-lto-support
fsfw/fsfw/pipeline/pr-development This commit looks good Details
eb0223bc51 Merge branch 'development' into mueller/add-lto-support
muellerr added 1 commit 2022-05-12 17:38:08 +02:00
fsfw/fsfw/pipeline/pr-development There was a failure building this commit Details
23c6145971
Merge remote-tracking branch 'origin/development' into mueller/refactor-logging-with-fmt
muellerr added 1 commit 2022-05-12 17:40:49 +02:00
fsfw/fsfw/pipeline/pr-development There was a failure building this commit Details
e2d3158506
remove v prefix from fmt version
muellerr added 1 commit 2022-05-12 18:59:56 +02:00
fsfw/fsfw/pipeline/pr-development This commit looks good Details
7ab617accb
rudimentary clion support
muellerr added 1 commit 2022-05-12 19:06:00 +02:00
fsfw/fsfw/pipeline/pr-development There was a failure building this commit Details
cdc431ebc5 Merge remote-tracking branch 'origin/development' into mueller/refactor-logging-with-fmt
muellerr added 1 commit 2022-05-13 13:17:10 +02:00
fsfw/fsfw/pipeline/pr-development There was a failure building this commit Details
cb9c1806ef
Merge remote-tracking branch 'origin/development' into mueller/refactor-logging-with-fmt
muellerr added 1 commit 2022-05-13 13:21:24 +02:00
fsfw/fsfw/pipeline/pr-development There was a failure building this commit Details
842f1b22af Merge branch 'development' into mueller/refactor-logging-with-fmt
muellerr added 1 commit 2022-05-16 14:41:25 +02:00
fsfw/fsfw/pipeline/pr-development There was a failure building this commit Details
8d966de735
Merge remote-tracking branch 'upstream/development' into mueller/refactor-logging-with-fmt
muellerr added 1 commit 2022-05-16 14:47:23 +02:00
fsfw/fsfw/pipeline/pr-development There was a failure building this commit Details
aea4e5d42c
resolve merge conflict
muellerr added 1 commit 2022-05-16 15:00:03 +02:00
muellerr commented 2022-05-18 12:59:22 +02:00
Author
Owner
Copy Link

I checked the code size when compiling the OBSW for FreeRTOS. It is significantly larger than the printf variant, at around 840kB when compiled with size optimization. It was planned to keep the OBSW image size below 840kB to allow booting from the NOR-Flash which would not be possible like this because the OBSW will grow.

I am not sure if it's really feasable to pull in big dependencies like this, especially when small bare metal code want to only use parts of the FSFW. The best solution might be to still keep supporting the traditional printf API by renaming FSFW_CPP_OSTREAM_ENABLED to FSFW_USE_FMT_LOG. Systems with a runtime like Embedded Linux or bare metal systems with flash sizes over 1MB can then use the FMT logger while smaller systems can use the printf API. One advantage of that would be that the API is really similar. Some suggestions/initial ideas on how to name the new API:

printf API

Without timestamp

INFO: sif::pinfo(...)
DEBUG: sif::pdebug(...)
WARNING: sif::pwarning(...)
ERROR: sif::perror(...)

With timestamp

Append _(s)t to the API calls shown above

With source file location

Append _s(t) to the API calls shown above

fmt API

Without timestamp

INFO: sif::finfo(...)
DEBUG: sif::fdebug(...)
WARNING: sif::fwarning(...)
ERROR: sif::ferror(...)

With timestamp

Append _t to the API calls shown above

Macro API

The macros will forwards all their arguments to either the printf API or to the
fmt API. Thanks to the similar API this is possible.

Without timestamp

INFO: FSFW_LOGI(...)
DEBUG: FSFW_LOGD(...)
WARNING: FSFW_LOGW(...)
ERROR: FSFW_LOGE(...)

With timestamp

Append T to the macro calls above

With source file location

Harcoded, enabled by default for DEBUG, WARNING and ERROR

Additional information

Coloring of the logs is optional.

I checked the code size when compiling the OBSW for FreeRTOS. It is significantly larger than the `printf` variant, at around 840kB when compiled with size optimization. It was planned to keep the OBSW image size below 840kB to allow booting from the NOR-Flash which would not be possible like this because the OBSW will grow. I am not sure if it's really feasable to pull in big dependencies like this, especially when small bare metal code want to only use parts of the FSFW. The best solution might be to still keep supporting the traditional `printf` API by renaming `FSFW_CPP_OSTREAM_ENABLED` to `FSFW_USE_FMT_LOG`. Systems with a runtime like Embedded Linux or bare metal systems with flash sizes over 1MB can then use the FMT logger while smaller systems can use the `printf` API. One advantage of that would be that the API is really similar. Some suggestions/initial ideas on how to name the new API: ### printf API #### Without timestamp INFO: `sif::pinfo(...)` DEBUG: `sif::pdebug(...)` WARNING: `sif::pwarning(...)` ERROR: `sif::perror(...)` #### With timestamp Append `_(s)t` to the API calls shown above #### With source file location Append `_s(t)` to the API calls shown above #### fmt API #### Without timestamp INFO: `sif::finfo(...)` DEBUG: `sif::fdebug(...)` WARNING: `sif::fwarning(...)` ERROR: `sif::ferror(...)` #### With timestamp Append `_t` to the API calls shown above ### Macro API The macros will forwards all their arguments to either the `printf` API or to the `fmt` API. Thanks to the similar API this is possible. #### Without timestamp INFO: `FSFW_LOGI(...)` DEBUG: `FSFW_LOGD(...)` WARNING: `FSFW_LOGW(...)` ERROR: `FSFW_LOGE(...)` #### With timestamp Append `T` to the macro calls above #### With source file location Harcoded, enabled by default for DEBUG, WARNING and ERROR ### Additional information Coloring of the logs is optional.
muellerr added 5 commits 2022-05-18 13:11:27 +02:00
fsfw/fsfw/pipeline/pr-development This commit looks good Details
618f76ae78
improved TcPacketPus API
fsfw/fsfw/pipeline/pr-development There was a failure building this commit Details
16f8262a79
starting a bit with event manage replacements
muellerr added 1 commit 2022-05-18 13:16:01 +02:00
fsfw/fsfw/pipeline/pr-development There was a failure building this commit Details
adcb646c9b
Merge remote-tracking branch 'origin/development' into mueller/refactor-logging-with-fmt
muellerr added 1 commit 2022-05-27 15:26:46 +02:00
fsfw/fsfw/pipeline/pr-development There was a failure building this commit Details
9f83894d4c
Merge remote-tracking branch 'origin/development' into mueller/refactor-logging-with-fmt
muellerr added 1 commit 2022-06-21 10:47:25 +02:00
fsfw/fsfw/pipeline/pr-development There was a failure building this commit Details
059e60cada
Merge remote-tracking branch 'origin/development' into mueller/refactor-logging-with-fmt
muellerr added 1 commit 2022-06-21 11:03:32 +02:00
muellerr added 1 commit 2022-06-21 11:14:54 +02:00
fsfw/fsfw/pipeline/pr-development There was a failure building this commit Details
42a1e784c0
logging fixes
muellerr added 1 commit 2022-07-18 08:53:48 +02:00
fsfw/fsfw/pipeline/pr-development There was a failure building this commit Details
a6ff3bb328
Merge remote-tracking branch 'origin/development' into mueller/refactor-logging-with-fmt
muellerr added 1 commit 2022-07-18 15:12:59 +02:00
fsfw/fsfw/pipeline/pr-development There was a failure building this commit Details
af7ebd3564 Merge branch 'development' into mueller/refactor-logging-with-fmt
mohr removed this from the v6.0.0 milestone 2023-02-09 13:34:36 +01:00
Some checks failed
fsfw/fsfw/pipeline/pr-development There was a failure building this commit
Details
This pull request has changes conflicting with the target branch.
  • CMakeLists.txt
  • src/fsfw/action/ActionHelper.cpp
  • src/fsfw/cfdp/CFDPHandler.cpp
  • src/fsfw/cfdp/pdu/EofPduDeserializer.cpp
  • src/fsfw/cfdp/pdu/VarLenField.cpp
  • src/fsfw/datapool/PoolReadGuard.h
  • src/fsfw/datapoollocal/HasLocalDataPoolIF.h
  • src/fsfw/datapoollocal/LocalDataPoolManager.cpp
  • src/fsfw/datapoollocal/LocalDataPoolManager.h
  • src/fsfw/datapoollocal/LocalPoolDataSetBase.cpp
You can also view command line instructions.

Step 1:

From your project repository, check out a new branch and test the changes.
git checkout -b mueller/refactor-logging-with-fmt development
git pull origin mueller/refactor-logging-with-fmt

Step 2:

Merge the changes and update on Gitea.
git checkout development
git merge --no-ff mueller/refactor-logging-with-fmt
git push origin development
Sign in to join this conversation.
Reviewers
No reviewers
mohr
gaisser
Labels
Clear labels   
API Change

Changes the API, so Users need to adapted

  
Breaking API Change

Changes the API in a non backwards compatible way

  
bug

Something is not working

  
build

Issues related to the buildsystem, including CI

  
cosmetics

Small change most users won't notice

  
Documentation

Improvement of code documentation

  
duplicate

This issue or pull request already exists

  
feature

New feature

  
help wanted

Need some help

  
hotfix

Quick or important fix that is merged directly into master, causing a new revision

  
invalid

#worksforme

  
question

More information is needed

  
Refactor

Contains a refactoring of some code

  
Tests

Issues related to testing code

  
wontfix

This won't be fixed

No Label
API Change
Breaking API Change
bug
build
cosmetics
Documentation
duplicate
feature
help wanted
hotfix
invalid
question
Refactor
Tests
wontfix
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: fsfw/fsfw#620
No description provided.
Delete Branch "mueller/refactor-logging-with-fmt"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?