Implementation of high speed point-to-point communication (app-to-app)?

[pmehradnia]

What are some different mechanisms to implement point-to-point communication to improve message passing performance? Mutex with OS_Queue is one option, but mqueues can be slow. What other solutions can be used to implement this?

[skliper]

Related:

• Add fixed size CCSDS packet option CF#193 (comment)
• Add user-space message queue library to the OSAL (GSFC DCR 22160) osal#73

@jphickey - did you have implementation options in mind?

[skliper]

In other words, I see MIDs as the efficient way of identifying CCSDS packets for routing, filtering, etc. I'm suggesting all that logic be separate from the actual P2P interface... the less it cares about the more efficient it can be.

[jphickey]

Actually, in a P2P context, I do not think CF would put data into CCSDS packets at all. The sender (CF) would know its a point to point link and it can skip all that extra work. The only reason I can see it still using CCSDS framing would be as a compatibility stop-gap measure in case one or the other side of a connection did not know it has a P2P link (but if that's the case then backpressure won't be possible either).

I was referring to MIDs here not for how the software uses it, but how a systems engineer uses it. In the concept of "this data exists in my system, how do I get it to another part of my system that is interested in it". In the case of CFDP PDUs, it's a matter of connecting CF to something that can send those PDUs to their next hop on the eventual path to their destination. The system in general just needs a way to connect the producer to the consumer, in an abstract way, that doesn't reinvent the wheel. SB currently does this via MIDs.a Perhaps it could (and maybe should?) use text-based names instead for ALL of this, it would be much more abstract. (i.e. even in a pub/sub context, instead of subscribing to SAMPLE_APP_HK_TLM_MID ... it could subscribe to cfe_sb://SAMPLE_APP:1/HK_TLM -- made to look URI-like).

In a P2P context, there is no routing on that hop, so CCSDS framing doesn't add much value. Furthermore, CCSDS has other problems in this context - a big one is that it insists that the data be labeled as "CMD" or "TLM" -- its a fixed binary bit that has to be set one way or the other - which may lead to misclassification/mishandling of the data - exactly the problem we need to avoid, as this is neither CMD nor TLM.

IMO the root of this problem is the fact that the system labels these CFDP PDUs as "Telemetry data" ... which by definition is historically fixed size (not even a terrible assumption to make, TLM is supposed to be fixed/consistently formatted messages after all). So for one hop to want to assign a "known size" to a given TLM MID is not wrong. What's wrong is the fact that it was labeled as TLM data in the first place (its not).

[jphickey]

To summarize - I would definitely be in favor of replacing numeric MIDs entirely (both for existing pub/sub as well as for future p2p) with a more abstract endpoint name like a CFE URI or similar. Those endpoint names can be looked up in a the mission data dictionary (because we have one, right?) that reveals the nature of the data (its size, frequency, nature of the data, its routing info i.e. the MID it is actually being published on). Such a lookup would only be done at time of subscription which is really just a start up thing, not for every message, of course.

In the meantime, without that separate lookup table, we've historically used numeric MID for that purpose, making that number do double-duty as both an administrative identifier and runtime identifier. The latter (runtime) is not necessary for P2P, but the former (admin) is.

[skliper]

Although I agree CF wouldn't have to put PDUs into CCSDS packets, something does. I see more negatives in pushing that work downstream than positives (one less header would be slightly less overhead in the queue but the extra logic downstream to handle the CF P2P link as "special" doesn't seem worth it). As long as it's CCSDS packets, everything downstream knows how to handle it using already established techniques, and as long as the downstream entities actually look at the size it's just like any other CCSDS packet when it comes to routing and there doesn't need to be any unique logic.

I still don't think the P2P interface software or system engineers need to care what MIDs are on the link. It's simply a queue the way I think of it. There's nothing that requires CCSDS packets to be fixed size other than historical systems that can't handle variable sized CCSDS packets. Although I do see the possible confusion in labeling PDU's as TLM since they can be received or sent I really don't see that as an issue... it's not an app command by any means, it's on a unique MID so I'm fine with calling it TLM. Other apps receive TLM and handle it appropriately (think DS, HS, HK, etc all receive tlm), I don't see why CF can't do the same. To every other entity but CF it's just a CCSDS packet with an MID that needs to be routed to the appropriate destination. For incoming the external link could simply filter on MID's (not caring if it's marked as TLM or CMD) and put it on the P2P to CF. Outgoing it gets the CCSDS packet from CF over the P2P and simply puts it on the external link. Much like every other CCSDS packet, only the endpoint cares if it's labeled CMD or TLM. Although FWIW there are implementations that may handle front end CMD verification... but as long as it's labeled TLM there's no impact.

Summary - Value of CCSDS prior to the P2P hop keeps the unique handling localized. If the other endpoint is just handling CCSDS packets impacts are minimized, then P2P becomes useful in many scenarios where efficiency is a priority over pub/sub. As soon as you start transmitting raw PDU's across an interface, anything that receives it is now required to know what a PDU is.

[skliper]

All good conversation for redesign options of CF and PDU handling, MIDs and CCSDS handling redesign, etc but maybe a good time to move those topics back to their more closely associated issues (maybe nasa/CF#193 and nasa/cFE#732?) Back to the P2P interface, my preference is to not bake it into SB and provide as either an extension of OS_Queue's or a separate library with a documented point to point restriction (design for efficiency and support the concept of back-pressure) that treats the data as opaque (size and address). Then it's generally applicable to a wide variety of use cases whatever path we go down for CF. Really could just be a user-space queue limited to P2P which could be a subset of or related to nasa/osal#73?