Storing and Transferring Data Larger Than 65536 bytes

[jwberck]

Currently, it seems like if you want to transfer more than 65536 bytes between apps, you need to roll your own system of writing to shared memory and managing access to it. Likewise, since 65536 is the max size of anything going through the software bus abstraction, you must write to disk with OSAL calls or FM instead of using DS to store files. Are there any common best practices or set of tooling here? It seems like another shared abstraction/tool is needed for larger data, which will become increasingly common. I understand the SB is trying to be like the CCSDS, so it seems we can't modify that. Are there any plans to capture this type of situation in CFS to prevent everyone from rolling their own solution to it?

[skliper]

See #425 as related
Also discussed in the context of CF in nasa/CF#130

Point to point queues have been discussed in the context of CF and large app transfers in the past, but I'm not aware of anyone actively working the issue. We do encourage contribution/collaboration in that area though since it is an identified need/use case, so rather than roll your own consider rolling one for the community!

[jwberck]

Makes sense, looking into FM more, it seems it can't really be used for large file storage without the large app transfer system. If we come up with some libraries for these, we'll try to push them up!

[skliper]

Could you elaborate on the use case? I think of FM more for file management (copy, move, rename, delete, concatenate, directory operations, etc) vs data transfer so I don't see how the CCSDS packet size limit is restricting these sorts of operations?

[jwberck]

Yeah that’s actually what I mean haha. Looking into it more I realized the use case you mention is the true nature of it. Seems like we need to roll our own or chop it up into SB sized packets and reconstruct it on the other end. I’m thinking the former is going to be our way forward. We’re thinking about having one library handling large data transfer between apps and one handling logging that data to the file system.

[skliper]

I did just stumble on nasa/osal#18, apparently OSAL at one point had a set of shared memory APIs... could be a way to share large data between apps if. Could be added as a library or PSP module, although according to the conversation on that issue an implementation was not found.

[rlourette]

I am considering writing a remote table app that will allow one to "open" tables on one CPU node and transfer them to the requesting node. I have SpaceWire in my system so RMAP can be used to physically move the data, but that is not applicable as a general solution. Thoughts?

[irowebbn]

#325 also discusses this question, although writing to intermediate files was suggested as a solution there, which James mentioned he specifically wanted to avoid.