Updates made as part of the LINKx PR series

[amirshehataornl]

As part of the LINKx development I have made some API changes. I would like to discuss these changes with the concerned parties to get alignment and move the changes along.

NOTE: I kept each of the API changes in its own PR so that they can be dealt with independently and can potentially land independently.

Address matching callback: #10039

Some providers (including CXI) do not always have the fi_addr_t to pass to the get_msg()/get_tag() calls. In order for them to do that then they will need to do a reverse lookup which is not efficient. I do not think the peer infrastructure should introduce this requirement. The general design of libfabric made it a point not to add this requirement as apparent by the need for FI_SOURCE. From the man page:

FI_SOURCE
Requests that the endpoint return source addressing data as part of its completion data. This capability only applies to connectionless endpoints. Note that returning source address information may require that the provider perform address translation and/or look-up based on data available in the underlying protocol in order to provide the requested data, which may adversely affect performance. The performance impact may be greater for address vectors of type FI_AV_TABLE.

Therefore, I propose the addition of an address matching callback which the parent provider would call. The idea here is that when the parent provider attempts to do address matching it can just use the native provider address to do the matching.

Add fi_peer_match to pass to the get_msg()/get_tag() callbacks: #10028

now that we need to pass more information in the get_msg()/get_tag() callbacks in order to be able to do proper address matching, it makes more sense to add a structure to pass to these callbacks. Currently I just add all the fields in the structure, but we can potentially unionize the structure in order to make it clearer what fields are being used. This pull request adds the fi_peer_match structure and makes the necessary updates in the code to accommodate this change.

Add memory registration callback to fi_ops_srx_peer: #10029

The CXI provider (and potentially other providers as well) register the receive buffers with the NIC on the receive path before data can be copied into these buffers. They then add the mapped addresses to the cache in order not to repeatedly do the expensive pinning/unpinning operations. This logic happens when the receive request is submitted. With the peer infrastructure this registration of the receive buffers is delayed until a message is processed by the provider and get_msg()/get_tag() is called. Through experimentation it has been observed that delaying the registration causes a significant performance hit. I suspect this is likely due to the delay incurred for pinning the memory in the NIC before the transfer into the buffers can occur. As opposed to having the buffers already pinned before the data arrives. Basically, from the perspective of the pinning operation, all messages are now dealt with as though they are unexpected.

This PR introduces a memory registration callback which the parent provider can use to register the memory with the core provider on the receive path, if such a callback is provided.

The caveat here is that this can only be done if the peer to receive from is known. So this approach is not done when the source address is FI_ADDR_UNSPEC.

Return the SRX peer callbacks to the parent provider: #10031

There are two reasons I think this change makes sense:

1. From a clarity perspective I think it makes sense for all the peer infrastructure to behave in a similar manner. In the shared completion queue case the core provider returns the peer callbacks to the parent provider. I think it is a good idea for both the completion queue and SRX to behave the same. I also believe the initial design was so.
2. Now that I added the memory registration callback to the peer SRX callbacks, I need the callbacks outside of the get_msg()/get_tag() path. This PR returns it, although I'm not entirely convinced that this is the best way to do it. And I'm not convinced that the current way SRX is handled during initialization makes sense. The code currently makes a copy of the SRX block which is passed by the parent provider and then updates the copy with the peer callbacks. I do not understand why we need to do this. The block which is passed down by the parent provider is owned by the parent provider and should not be copied. The parent provider should guarantee that this block remains viable as long as the peer provider is using it. This is the same strategy taken by the shared completion queues. If we do that, then the peer provider can add its callbacks to the block which is owned by the parent provider. This allows the parent provider to get access to the peer callbacks in a straightforward manner.

ADD FI_PEER as a capability bit: #10030

LINKx can only link providers which support the peer infrastructure. I added the FI_PEER bit as a capability, this way when the LINKx provider attempts to link providers it is able to check whether they support the peer infrastructure

[aingerson]

@amirshehataornl Sorry for the late reply. Got busy. Let's unpack this one by one:

1. Address matching callback
   Can you provide an example of the lookup and how it would be passed around? If the peers are totally opaque to each other, how do we get agreement from peers that we won't use the same matching method or grab the wrong peer? Would the flow be that when a peer calls get_msg, the owner has to lookup who called get_msg and pick the correct callback? On the receive, I'm assuming the owner would detect who the buffer is for and separate the posts by which peer to match to?

2. Peer match
   I think that sounds reasonable to me. I would like to clearly define which ones are needed for which functions and who sets them. Right now there is a discrepancy as to who sets the fields, specifically the size parameter. I think it would be good to meet and discuss the flow we want to define for the initialization and usage of the peer_rx_entry fields. My gut says that the owner should initialize the fields and the peer shouldn't be able to modify them (except the peer context) but I would like to hear what folks have to say.

3. Memory registration callback
   Also open to this but would like to discuss the proper defined flow and potentially peer MR registration API options to solve some of the duplicate memory registration happening in the peer path

4. srx op copying
   Not sure I follow the issue here. As it is now, the owner does have access to the peer ops since it does the copying. The copying is really just for the owner ops so that the peer has a reference to the owner ops but can set the peer ops so multiple peers can be used. From the owner's point of view, when it initializes the peer resources (CQ, SRX, counter), it will still get a new fid back from the peer that the peer owns. That fid is unique to that peer's shared resource and the owner can access the peer ops through that fid. Can you elaborate a bit more on the issue you're seeing?

5. FI_PEER capability
   Also seems reasonable to me but we also need to define what that means obviously. Don't see an issue with that though.

@shijin-aws Do you have thoughts on these? I would like to have a meeting with you both (and anyone else you think that should be a part of it of course) about all of these items.

[amirshehataornl]

Sorry for the delay in response.

1. The way I implemented this is by adding an address matching function in the peer_ops. When get_tag()/get_msg() is called I'm able to lookup the peer_ops of the calling provider, look up the address matching function and call it. The provider passes contextual data (typically the native address which was received). LINKx would then call back into the provider using the address matching function to perform the address matching. This avoids the reverse lookup from native address to fi_addr_t.
2. I'm okay with the owner taking responsibility for the owner provided structures.
3. Meet and discuss.
4. The main point I'm making here is the owner provider needs to be able to call the peer_ops outside get_msg()/get_tag() calls. As far as I understood the code that's currently not possible.

I think it would be good to meet and discuss these points.