Introducing Multi-Path DMA Support for mlx5 RDMA Driver #27
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tdavenvidia
requested review from
jamieNguyenNVIDIA,
nvmochs,
KobaKoNvidia and
clsotog
jamieNguyenNVIDIA
requested changes
Oct 17, 2024
There was a problem hiding this comment.
Can you add backport/cherry-pick information to each commit?
Something like:
(cherry picked from commit #### linux)
or
(backported from commit #### linux)
From what I can tell:
(1/8) picked cleanly
(2/8) picked cleanly
(3/8) did not pick cleanly, but backport looked straightforward
(4/8) picked cleanly
(5/8) picked cleanly
(6/8) picked cleanly
(7/8) did not pick cleanly, but backport seemed okay despite the large commit
(8/8) picked cleanly
The patches themselves look good outside of the missing cherry-pick/backport information.
|
The patches looks ok for me. |
tdavenvidia
force-pushed
the
PR-multipath-dma-for-adv-next
branch
from
570f73a
to
d5c66ca
Compare
|
Added cherry-pick/backported info. |
|
I confirmed Jamie's assessment, verified that there are no upstream fixes for these commits, and that the SHAs in the pick tags matches upstream. Acked-by: Matthew R. Ochs [email protected] |
There was a problem hiding this comment.
It looks like all of these are in Torvalds' tree now, so I would call out those commits instead of the rdma/for-next ones:
ec7ad6530909 RDMA/mlx5: Introduce GET_DATA_DIRECT_SYSFS_PATH ioctl
de8f847a5114 RDMA/mlx5: Add support for DMABUF MR registrations with Data-direct
3aa73c6b795b RDMA: Pass uverbs_attr_bundle as part of '.reg_user_mr_dmabuf' API
253c61dc256b RDMA/umem: Introduce an option to revoke DMABUF umem
682358fd35de RDMA/umem: Add support for creating pinned DMABUF umem with a given dma device
2e8e631d7a41 RDMA/mlx5: Add the initialization flow to utilize the 'data direct' device
6910e3660d86 RDMA/mlx5: Introduce the 'data direct' driver
a18eb864019e Introducing Multi-Path DMA Support for mlx5 RDMA Driver
...
c772a2c69018 net/mlx5: Add IFC related stuff for data direct
EDIT: the commit SHAs seem to match the ones in Torvalds' tree, so I think it's just a matter of s/rdma:for-next/linux/g.
nvmochs
approved these changes
Oct 18, 2024
|
IMO 's/rdma:for-next/linux/g' is really not needed, but I am doing it now :) |
added 8 commits
October 18, 2024 19:53
Add IFC related stuff for data direct. Signed-off-by: Yishai Hadas <[email protected]> Link: https://patch.msgid.link/82da7f578a567909bb5858a64ba844fe4cc298fa.1722512548.git.leon@kernel.org Signed-off-by: Leon Romanovsky <[email protected]> (cherry-picked from commit c772a2c linux) Signed-off-by: Tushar Dave <[email protected]>
Introduce the 'data direct' driver for a ConnectX-8 Data Direct device. The 'data direct' driver functions as the affiliated DMA device for one or more capable mlx5_ib devices. This DMA device, as the name suggests, is used exclusively for DMA operations. It can be considered a DMA engine managed by a PF/VF, lacking network capabilities and having minimal overall capabilities. Consequently, the DMA NIC PF will not be exposed to or directly used by software applications. The driver will not have any direct interface or interaction with the firmware (no command interface, no capabilities, etc.). It will operate solely over PCI to enable its DMA functionality. Registration and un-registration of the driver are handled as part of the mlx5_ib initialization and exit processes, as the mlx5_ib devices will effectively be its clients. The driver will serve as the DMA device for accessing another PCI device to achieve optimal performance (both on the same NUMA node, P2P access, etc.). Upon probing, it will read its VUID over PCI to handle mlx5_ib device registrations with the same VUID. Upon removal, it will notify its clients to allow them to clean up the resources that were mmaped with its DMA device. Signed-off-by: Yishai Hadas <[email protected]> Link: https://patch.msgid.link/b77edecfd476c3f445da96ab6aef499ae47b2829.1722512548.git.leon@kernel.org Signed-off-by: Leon Romanovsky <[email protected]> (cherry-picked from commit 6910e36 linux) Signed-off-by: Tushar Dave <[email protected]>
…evice Add the NET device initialization flow to utilize the 'data direct' device. When a NET mlx5_ib device is capable of 'data direct', the following sequence of actions will occur: - Find its affiliated 'data direct' VUID via a firmware command. - Create its own private PD and 'data direct' mkey. - Register to be notified when its 'data direct' driver is probed or removed. The DMA device of the affiliated 'data direct' device, including the private PD and the 'data direct' mkey, will be used later during MR registrations that request the data direct functionality. Signed-off-by: Yishai Hadas <[email protected]> Link: https://patch.msgid.link/b11fa87b2a65bce4db8d40341bb6cee490fa4d06.1722512548.git.leon@kernel.org Signed-off-by: Leon Romanovsky <[email protected]> (backported from commit 2e8e631 linux) [tdave: Resolve conflict in main.c and mlx5_ib.h] Signed-off-by: Tushar Dave <[email protected]>
…ma device Add support for creating pinned DMABUF umem with a specified DMA device instead of the DMA device of the given IB device. This API will be utilized in the upcoming patches of the series when multiple path DMAs are implemented. Signed-off-by: Yishai Hadas <[email protected]> Link: https://patch.msgid.link/038aad36a43797e5591b20ba81051fc5758124f9.1722512548.git.leon@kernel.org Signed-off-by: Leon Romanovsky <[email protected]> (cherry-picked from commit 682358f linux) Signed-off-by: Tushar Dave <[email protected]>
Introduce an option to revoke DMABUF umem. This option will retain the umem allocation while revoking its DMA mapping. Furthermore, any subsequent attempts to map the pages should fail once the umem has been revoked. This functionality will be utilized in the upcoming patches in the series, where we aim to delay umem deallocation until the mkey deregistration. However, we must unmap its pages immediately. Signed-off-by: Yishai Hadas <[email protected]> Link: https://patch.msgid.link/a38270f2fe4a194868ca2312f4c1c760e51bcbff.1722512548.git.leon@kernel.org Signed-off-by: Leon Romanovsky <[email protected]> (cherry-picked from commit 253c61d linux) Signed-off-by: Tushar Dave <[email protected]>
Pass uverbs_attr_bundle as part of '.reg_user_mr_dmabuf' API instead of udata. This enables passing some new ioctl attributes to the drivers, as will be introduced in the next patches for mlx5 driver. Change the involved drivers accordingly. Signed-off-by: Yishai Hadas <[email protected]> Link: https://patch.msgid.link/9a25b2fc02443f7c36c2d93499ae25252b6afd40.1722512548.git.leon@kernel.org Signed-off-by: Leon Romanovsky <[email protected]> (cherry-picked from commit 3aa73c6 linux) Signed-off-by: Tushar Dave <[email protected]>
Add support for DMABUF MR registrations with Data-direct device. Upon userspace calling to register a DMABUF MR with the data direct bit set, the below algorithm will be followed. 1) Obtain a pinned DMABUF umem from the IB core using the user input parameters (FD, offset, length) and the DMA PF device. The DMA PF device is needed to allow the IOMMU to enable the DMA PF to access the user buffer over PCI. 2) Create a KSM MKEY by setting its entries according to the user buffer VA to IOVA mapping, with the MKEY being the data direct device-crossed MKEY. This KSM MKEY is umrable and will be used as part of the MR cache. The PD for creating it is the internal device 'data direct' kernel one. 3) Create a crossing MKEY that points to the KSM MKEY using the crossing access mode. 4) Manage the KSM MKEY by adding it to a list of 'data direct' MKEYs managed on the mlx5_ib device. 5) Return the crossing MKEY to the user, created with its supplied PD. Upon DMA PF unbind flow, the driver will revoke the KSM entries. The final deregistration will occur under the hood once the application deregisters its MKEY. Notes: - This version supports only the PINNED UMEM mode, so there is no dependency on ODP. - The IOVA supplied by the application must be system page aligned due to HW translations of KSM. - The crossing MKEY will not be umrable or part of the MR cache, as we cannot change its crossed (i.e. KSM) MKEY over UMR. Signed-off-by: Yishai Hadas <[email protected]> Link: https://patch.msgid.link/1f99d8020ed540d9702b9e2252a145a439609ba6.1722512548.git.leon@kernel.org Signed-off-by: Leon Romanovsky <[email protected]> (backported from commit de8f847 linux) [tdave: resolve conflict in main.c, mr.c and mlx5_user_ioctl_cmds.h , also rework reg_user_mr_dmabuf() to sync with recent upstream rdma changes] Signed-off-by: Tushar Dave <[email protected]>
Introduce the 'GET_DATA_DIRECT_SYSFS_PATH' ioctl to return the sysfs path of the affiliated 'data direct' device for a given device. Signed-off-by: Yishai Hadas <[email protected]> Link: https://patch.msgid.link/403745463e0ef52adbef681ff09aa6a29a756352.1722512548.git.leon@kernel.org Signed-off-by: Leon Romanovsky <[email protected]> (cherry-picked from commit ec7ad65 linux) Signed-off-by: Tushar Dave <[email protected]>
tdavenvidia
force-pushed
the
PR-multipath-dma-for-adv-next
branch
from
d5c66ca
to
9734646
Compare
jamieNguyenNVIDIA
approved these changes
Oct 18, 2024
clsotog
approved these changes
Oct 18, 2024
|
Acked-by: Jamie Nguyen [email protected] (Not sure if you still needed mine since Carol gave hers above) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This patch series cherry-pick/backport multipath-DMA Support patch series from upstream, https://lore.kernel.org/all/[email protected]/