Custom Packet Specification and Serialization Strategy #92

New Issue

muellerr · 2024-01-26T12:43:02+01:00

muellerr commented

2024-01-26 12:43:02 +01:00

There is a huge information duplication between ground and on-board software systems because of the need to transport a large number of custom data chunks with various custom formats "over the wire". For regular software systems, a lot of solutions were invented to solve this issue in an efficient manner. Some analysis was conducted on serialization strategies:

Exchange of Plain-Old Datastructures (POD) using tightly packed C types with network endianness. This is what is done for EIVE, FLP and SOURCE. Is it a good idea? Not necessarily, if enough resources are available. The format is not self-describing, and parser/marhsalling code is often written 2-3 times. However, this is an extremely memory efficient format, and the implementation of the serializer code is realtively easy for common types.
Use a self-describing format like JSON. Rust offers crates like serde which make serialization trivial (and with trivial, I really mean REALLY fucking easy). It is even human-readable. It probably has the most awful memory footprint of all formats. However,
gziped JSON strings could be a really nice solution, especially when a lot of memory bandwith is available. Alternatively, formats like MessagePack or BSON could be used to solve the memory bandwidth issue.
Use a cross-platform serialization format like protobuf. This is still a binary serialization format, but there is an external description language to describe the data being exchanged. This could be an especially nice solution if a full standard runtime is available.

Another issue which is also related is the information duplication between ground data systems and the on-board software. The ground system oftentimes needs some high-level information of the data being processed, independently of the used serialization format. It might use this information to submit the received data to a database using an ORM system. One possible approach would be to use Rust macros capabilities to attach necessary metadata to Rust structures.

For example the following structure for MGM data is shown, with some added metadata for MIB purposes:

#[mib_provider, Serialize, Deserialize...]
struct MyMgmData {
    #[mib(desc="X axis in nT", unit="nT")]
    x_nano_tesla: f32,
    #[mib(desc="Y axis in nT", unit="nT")]
    y_nano_tesla: f32,
    #[mib(desc="Y axis in nT", unit="nT")]
    z_nano_tesla: f32
}

This could for example generate special (static) output structure like this:

#[derive(Debug, Copy, Clone)]
struct MibParameter {
    name: &'static str
    type: &'static str,
    description: &'static str,
    unit: &'static str,
}

pub const MY_MGM_DATA_X_NANO_TESLA: MibParameter = MibParameter {
     name: "x_nano_tesla",
     type: "F32",
     description: "X axis in nT",
     unit: "nT"
}

....

pub const MyMgmDataMib: &[MibParameter] = {
    MY_MGM_DATA_X_NANO_TESLA,
    MY_MGM_DATA_Y_NANO_TESLA,
    MY_MGM_DATA_Z_NANO_TESLA
};

These output structures could be used to generate the data format that the ground system requires automatically.. This would be an approach where we don't need auto-generation and the OBSW programmer can still write the data structures however they see fit.

There is a huge information duplication between ground and on-board software systems because of the need to transport a large number of custom data chunks with various custom formats "over the wire". For regular software systems, a lot of solutions were invented to solve this issue in an efficient manner. Some analysis was conducted on serialization strategies: 1. Exchange of Plain-Old Datastructures (POD) using tightly packed C types with network endianness. This is what is done for EIVE, FLP and SOURCE. Is it a good idea? Not necessarily, if enough resources are available. The format is not self-describing, and parser/marhsalling code is often written 2-3 times. However, this is an extremely memory efficient format, and the implementation of the serializer code is realtively easy for common types. 2. Use a self-describing format like JSON. Rust offers crates like `serde` which make serialization trivial (and with trivial, I really mean REALLY fucking easy). It is even human-readable. It probably has the most awful memory footprint of all formats. However, gziped JSON strings could be a really nice solution, especially when a lot of memory bandwith is available. Alternatively, formats like MessagePack or BSON could be used to solve the memory bandwidth issue. 3. Use a cross-platform serialization format like `protobuf`. This is still a binary serialization format, but there is an external description language to describe the data being exchanged. This could be an especially nice solution if a full standard runtime is available. Another issue which is also related is the information duplication between ground data systems and the on-board software. The ground system oftentimes needs some high-level information of the data being processed, independently of the used serialization format. It might use this information to submit the received data to a database using an ORM system. One possible approach would be to use Rust macros capabilities to attach necessary metadata to Rust structures. For example the following structure for MGM data is shown, with some added metadata for MIB purposes: ```rs #[mib_provider, Serialize, Deserialize...] struct MyMgmData { #[mib(desc="X axis in nT", unit="nT")] x_nano_tesla: f32, #[mib(desc="Y axis in nT", unit="nT")] y_nano_tesla: f32, #[mib(desc="Y axis in nT", unit="nT")] z_nano_tesla: f32 } ``` This could for example generate special (static) output structure like this: ```rs #[derive(Debug, Copy, Clone)] struct MibParameter { name: &'static str type: &'static str, description: &'static str, unit: &'static str, } pub const MY_MGM_DATA_X_NANO_TESLA: MibParameter = MibParameter { name: "x_nano_tesla", type: "F32", description: "X axis in nT", unit: "nT" } .... pub const MyMgmDataMib: &[MibParameter] = { MY_MGM_DATA_X_NANO_TESLA, MY_MGM_DATA_Y_NANO_TESLA, MY_MGM_DATA_Z_NANO_TESLA }; ``` These output structures could be used to generate the data format that the ground system requires automatically.. This would be an approach where we don't need auto-generation and the OBSW programmer can still write the data structures however they see fit.

muellerr added the

concepts

label 2024-01-26 16:58:59 +01:00

muellerr added this to the satrs v0.1.0 milestone 2024-01-26 17:08:45 +01:00

muellerr changed title from ~~Generating Mission Information Base (MIB) information~~ to Mission Information Base (MIB) information and Custom Packet Specification

2024-01-26 17:54:40 +01:00

muellerr changed title from ~~Mission Information Base (MIB) information and Custom Packet Specification~~ to Custom Packet Specification

2024-01-26 17:55:17 +01:00

muellerr changed title from ~~Custom Packet Specification~~ to Custom Packet Specification and Serialization Strategy

2024-01-26 17:56:44 +01:00

Sign in to join this conversation.