On-Demand Paging usage specified by the application

[swelch]

While the FI_MR_ALLOCATED mode bit indicates that the provider requires that registered memory regions must be backed by physical memory pages at the time the registration call is made, it is not clear to me in libfabric man pages how ODP optional usage should be indicated for a provider that supports it. For instance, if a provider performs RDMA and can be configured to support ODP, how should this be selected by the application in a libfabric common way. My assumption is that the application and provider can let the presence of the FI_MR_ALLOCATED mr_mode bit select whether ODP is desired or not. To support this case the provider may export multiple fi_info per domain, one with FI_MR_ALLOCATED MR mode bit set and one without it set. The application would select the correct fi_info based on their mr_mode hints they provide.

It appears the existing Verbs provider code requires that an undocumented environment variable "FI_VERBS_USE_ODP" be set to enable ODP, that the verbs extended query capabilities API is supported, that the verbs hardware supports ODP, and that the registration is for system memory. If this is true it will pass the appropriate attribute to the rdma-core. Independent of these settings it still requires FI_MR_ALLOCATED mr_mode bit.

@shefty - Thoughts on the correct usage and/or the best way an application can select that it wants to use ODP with an RDMA provider that supports ODP?

[shefty]

I would start by defining how ODP is being used. I believe the verbs provider enables it through an environment variable because having it enabled gives lower performance than some alternatives.

For buffers that are only accessible locally (e.g. send and receive buffers), ODP could eliminate any MR cache. The FI_MR_LOCAL bit could be cleared for this use case, with memory registration hidden from the app completely.

If we consider remotely accessed buffers (e.g. remote RMA read/write), I agree ODP aligns with FI_MR_ALLOCATED. This also ties into the MR cache and invalidating entries.

I would like to leave it up to the provider to determine whether enabling ODP provides better performance, versus using a MR cache. I don't think we get away from having some sort of environment variable to specify whether using ODP and/or the MR cache is acceptable from a security standpoint.

[swelch]

@shefty - This seems reasonable. Currently we export and fi_info with FI_MR_ALLOCATED in the mr_mode before an fi_info without FI_MR_ALLOCATED since non-ODP would be preferred. The thought was that fi_info query without hints would display both fi_info making it clear both are supported. However, using an environment variable to turn on ODP and control not setting FI_MR_ALLOCATED in the mr_mode seems like a suitable option as well. I'll discuss internally and migrate to that usage if possible.

[shefty]

We should have a broader discussion, maybe as part of 2.0, about whether the existence of a MR cache should be exposed to the application through the API.

[chuckfossen]

ODP presentation for OFIWG.pdf
I updated the presentation based on feedback from today's meeting. The section on FI_MR_LOCAL updates may need some more refinement.

[iziemba]

If OS triggers memory migration, there is nothing the provider can do without ODP. This is why, in HPC, things which cause the OS to migrate memory (THP, NUMA balancing, etc...) are explicitly disabled. If ODP is disabled and memory migration happens, depending on provider/driver MR implementation, either RDMA operations will unexpectedly fail or data corruption will occur (stale pages will be referenced).

We now have hardware platforms where host <-> GPU memory migration is supported for malloc() buffers and users are being encouraged to use this functionality. Specifically, HPE stakeholders are asking for RDMA and migratable GPU memory support. From a CXI provider point of view, this requires ODP to avoid unexpected RDMA failure.

But, if I am an app/middleware and running on hardware platform where host <-> GPU memory migration may happen, I need to know if the provider can handle to this avoid RDMA operation failure or data corruption.

In addition, updating FI_MR_ALLOCATED = 1 documentation to state that page migration is supported seems wrong because this now forces providers to support ODP or the equivalent.

[shefty]

We need to define something actionable on the part of the app. Example:

FI_MR_BIT_5
When set, the memory region must be backed by physical pages when registered.
While the region is registered, the app must not free or remap the region.

The action here is that the app must call malloc(), calloc(), mmap(), cudaMalloc(), or some other memory allocation call prior to registering the memory. After calling register, it cannot call free(), munmap(), cudaFree(), etc. without first unregistering the region.

If FI_MR_BIT_5 is unset (0), the above constraints are lifted. You seem to be asking to expand the meaning to include:

• The app can change the VA to PA mapping at any time, but only within the same memory domain. That is, host pages may be replaced with other host pages.
• Or the bit only applies to host memory regions.

Basically, the VA to PA can change, but you don't want that change to mean that the pages can switch between the host and a device.

And we need to be clear here. The VA to PA mapping is completely outside the control of the app. The OS can change the mapping at any time. The restriction on the app is that it won't invoke calls that make this likely to happen. It's that restriction that enables the provider to work with the OS to avoid it changing the mapping. Meaning, the provider pins the memory, so the PA is protected from being reused. The app side of the bargain is that it will keep its VA valid.

Now, with automatic page migration to/from device memory, you want to expose this as:

FI_MR_BIT_12
When set, the app does... **what**?

The point of exposing this flag to the app is for the app to dynamically change its execution. I have no idea what the app is supposed to do differently.

From the viewpoint of the app, page pinning and ODP aren't relevant. What matters is the behavior that the app is supposed to comply with.

[iziemba]

The action here is that the app must call malloc(), calloc(), mmap(), cudaMalloc(), or some other memory allocation call prior to registering the memory. After calling register, it cannot call free(), munmap(), cudaFree(), etc. without first unregistering the region.

Yes. I would like to see FI_MR_ALLOCATED = 1 be updated with this information. This will make it clear to apps that if this is not set, I can register large VA range and mmap/munmap within it and the MR is still valid. This addresses one of HPEs use cases.

If FI_MR_ALLOCATED = 0, this would mean provider can support VA to PA changes. This opens the door for page migration not only within the same memory domain, but across memory domains. Seems like we could reuse FI_MR_ALLOCATED = 0 for migration, correct? If so, I do not see a need for FI_MR_BIT_12.

In practice, most DMA providers will probably hook into Linux kernel HMM to implement migration. While HMM should always work for FI_HMEM_SYSTEM, whether or not it works for other devices is vendor hardware and software dependent. Thinking out loud, if we had a bit mask denoting for which devices FI_HMEM_SYSTEM can migrate to, apps can change their behavior. For example, if I know memory can migrate to FI_HMEM_CUDA, I'll use the malloc() VA in my GPU kernel. If not, I know I need to allocate GPU device memory.

[shefty]

I'm fine clarifying the definition of FI_MR_ALLOCATED. That's not a problem as long as it follows its original intent.

I'm even okay having some other bit indicate something related to device migration. But I don't know how to define such a bit. HMM migrates the pages without the app being aware, so I don't know how the app can take steps to prevent it. Maybe it's as simple as saying that all device memory must use some GPU specific function, and migration will never automatically occur.

[iziemba]

The problem here is some HW platforms may or may not support host to GPU migration with system memory. How is an app suppose to know this? Framing the question like this leads me to think that this is not a libfabric problem if we state that FI_MR_ALLOCATED = 0 allows system migration. Where this memory migrates to may not be a provider concern. For example, TCP provider should still be able to reference the VA and OS takes care of any migration. Similar with with verbs and CXI with ODP and HMM. If this is the case, we may not need a libfabric device migration bit either.

[iziemba]

I have opened up #10062 for this.