core: Introduce MR with dynamic keys #10317
Conversation
Memory registration consists of two parts: map/pin the memory for local access and export with a key for remote access. The first part is usually heavyweight and requries kernel involvement. The second part is less expensive and can be further separated into key allocation and key assignment. Key allocation may needs kernel involvement, but key assignment can be done in user space. This leads to the concept of MR with dynamic key. Dynamic key allows an MR be exported as different keys for different target, with the option to expose different address range and access rights, all without the overhead of registering the MR multiple times in the standard way. A new flag FI_MR_DYNAMIC_KEY is introduces to be used with memory registration calls. This flag also serves as a secondary capbility. New API functions are added to allocate / assign / revoke keys. Another flag FI_MR_SINGLE_USE is added for single use key assignment. Signed-off-by: Jianxin Xiong <[email protected]>
| *FI_MR_DYNAMIC_KEY* | ||
| : This flag indicated that the memory region will be associated with dynamic | ||
| keys. A key is assigned to the memory region using fi_mr_assign_key and is | ||
| removed from the memory region fi_mr_revoke__key. Valid keys are allocated |
There was a problem hiding this comment.
nit: extra '_' between revoke and key
| @@ -158,6 +158,8 @@ typedef struct fid *fid_t; | |||
| #define FI_MATCH_COMPLETE (1ULL << 31) | |||
|
|
|||
| #define FI_PEER_TRANSFER (1ULL << 36) | |||
| #define FI_MR_SINGLE_USE (1ULL << 38) | |||
There was a problem hiding this comment.
The global flag space is constrained. Please move this into fi_domain.h as a memory registration flag. The FI_MR_DMABUF flag below probably should have been located there as well.
| @@ -183,6 +183,21 @@ struct fi_mr_modify { | |||
| struct fi_mr_attr attr; | |||
| }; | |||
|
|
|||
| struct fi_mr_alloc_keys { | |||
| size_t count; | |||
| uint64_t *keys; | |||
There was a problem hiding this comment.
This assumes that the provider must support 64-bit keys, which may not be the case. It isn't for all existing providers.
|
This change is kind of like memory windows, but not. It requires the provider support 64-bit keys (or smaller). And it's actually adding 2 concepts together. FI_MR_SINGLE_USE is really independent and could apply to existing registrations. There's no way for a provider to indicate that it supports single use regions, outside of just trying the flag in a call and seeing if it works. I don't know if this should be a domain capability, but maybe... The dynamic keys are basically trying to create a new MR object that references the same pinned pages. That's equivalent to a memory window. However, MWs are created by posting to a QP/EP, whereas, this is a MR/domain operation. In either case, the user should be given a fid_mr structure here, not a u64 key. That allows integration with the other fi_mr calls (map, unmap, bind, refresh, enable). This is probably doable by adding fid_mr to the fi_mr_attr, uhm, somewhere. The user can just create/destroy the extra MRs through the existing calls, with the same capabilities/restrictions that the provider has for other MRs (user or provider selected keys). I don't know if we would actually need a new capability in the latter case. A provider could always perform a second registration, in which case, saving on the page pinning is simply an optimization. |
|
@shefty Thanks for the feedback. Yes, single use and dynamic key assignment did come as two separate issues but I combined them here with the assumption that single use could be simpler with the key instead of the entire MR. I agree that this mostly equivalent to memory windows. One consideration about the bulk key allocations is that it may have the advantage of avoiding kernel involvement when the key is assigned (in case the provider doesn't allow user selected keys). But how much can be gained from that is unclear. I am going to rework the patch, maybe along the line of getting closer to a MW-like approach. |
|
Bulk key allocation could be defined as an attribute when the original MR is created. I agree that a single use MR is less useful than a single use window/key. |
Memory registration consists of two parts: map/pin the memory for local access and export with a key for remote access. The first part is usually heavyweight and requries kernel involvement. The second part is less expensive and can be further separated into key allocation and key assignment. Key allocation may needs kernel involvement, but key assignment can be done in user space. This leads to the concept of MR with dynamic key.
Dynamic key allows an MR be exported as different keys for different target, with the option to expose different address range and access rights, all without the overhead of registering the MR multiple times in the standard way.
A new flag FI_MR_DYNAMIC_KEY is introduces to be used with memory registration calls. This flag also serves as a secondary capbility.
New API functions are added to allocate / assign / revoke keys.
Another flag FI_MR_SINGLE_USE is added for single use key assignment.