Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SYCL][CUDA][PI][runtime][ABI-break] Add support for multi-device context #6446

Closed
wants to merge 50 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
82b8bb9
cuda plugin changes
t4c1 May 16, 2022
02ceed2
modification to PI interface and runtime functions
t4c1 May 20, 2022
58591b2
Merge branch 'sycl' into multi_device_context
t4c1 May 23, 2022
6da5ea9
revert CUDA part of "[SYCL][CUDA][HIP] Report every device in its own…
t4c1 May 23, 2022
3c44f17
handle multiple contexts in program and kernel and memory transfers i…
t4c1 May 24, 2022
70c049c
fix memory transfers between devices
t4c1 May 27, 2022
7215c67
Merge branch 'sycl' into multi_device_context
t4c1 Jun 27, 2022
81ef2cf
enable multiple devioces in one context
t4c1 Jul 6, 2022
47a8200
format
t4c1 Jul 6, 2022
522a246
Merge branch 'sycl' into multi_device_context
t4c1 Jul 6, 2022
321829a
fix tests
t4c1 Jul 6, 2022
0bc7d62
Fix typo
joeatodd Jul 7, 2022
e014f8b
addressed comments
t4c1 Jul 7, 2022
301bd66
removed commenteed code
t4c1 Jul 7, 2022
5bdaef7
addressed review comments
t4c1 Jul 14, 2022
dbefec7
Merge branch 'sycl' into multi_device_context
t4c1 Jul 15, 2022
ec70ee0
format
t4c1 Jul 15, 2022
1d6198e
manual format
t4c1 Jul 15, 2022
a3761c0
fix pi_esimd_emulator
t4c1 Jul 15, 2022
ed57c83
actually fix pi_esimd_emulator
t4c1 Jul 18, 2022
791c8b5
fix for subdevice related tests
t4c1 Jul 19, 2022
3cb9042
format
t4c1 Jul 19, 2022
cd3c827
Merge branch 'sycl' into multi_device_context
t4c1 Aug 18, 2022
6a1bf98
format
t4c1 Aug 18, 2022
938bbe0
added querry for plugin memory connection
t4c1 Aug 18, 2022
d918d26
fix pi abi checks
t4c1 Aug 18, 2022
c2d1c6c
Apply suggestions to use ARG_UNUSED in pi_esimd_emulator.cpp from cod…
t4c1 Sep 8, 2022
7aeac7a
Merge remote-tracking branch 'origin/sycl' into HEAD
npmiller Nov 9, 2022
cc63a08
Fix PiMock after merging
npmiller Nov 9, 2022
62e1751
Fix ABI test
npmiller Nov 10, 2022
c8042bd
Cleanup memory connection interface
npmiller Nov 9, 2022
61385ce
Fix formatting
npmiller Nov 11, 2022
e54d662
Fix assert test
npmiller Nov 11, 2022
ac679f0
Fix ABI break for allocateMem
npmiller Nov 14, 2022
09560ae
Fix device_index function
npmiller Nov 14, 2022
50314ea
Refactor graph builder
npmiller Nov 15, 2022
341a4b4
Revert "Refactor graph builder"
npmiller Nov 16, 2022
11934a3
Merge remote-tracking branch 'origin/sycl' into multi_device_context
npmiller Jan 24, 2023
a1c0111
Fix merge conflicts
npmiller Jan 24, 2023
76d01d5
Merge remote-tracking branch 'origin/sycl' into multi_device_context
npmiller Jan 27, 2023
8f4c0ce
[SYCL][CUDA] Fix 2D USM active context
npmiller Jan 27, 2023
e06c597
[SYCL][CUDA] Clarify CUDA context usages
npmiller Jan 30, 2023
25dd438
[SYCL][CUDA] Keep track of devices in PI allocations
npmiller Feb 10, 2023
9179ee9
Merge remote-tracking branch 'origin/sycl' into multi_device_context
npmiller Feb 10, 2023
7e61df9
[SYCL][CUDA] Fix merge issues
npmiller Feb 10, 2023
fefc782
Merge remote-tracking branch 'upstream/sycl' into multi_device_context
npmiller Feb 24, 2023
d69134c
Fix PI cuda tests for multi-device context
npmiller Feb 24, 2023
5d4c941
Merge remote-tracking branch 'upstream/sycl' into multi_device_context
npmiller Apr 5, 2023
c5d6a0e
[SYCL] Update tests for multi-device context
npmiller Apr 5, 2023
deba394
[SYCL] Fix linting issues
npmiller Apr 5, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions sycl/include/sycl/detail/pi.def
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,8 @@ _PI_API(piextUSMGetMemAllocInfo)
_PI_API(piextEnqueueReadHostPipe)
_PI_API(piextEnqueueWriteHostPipe)

_PI_API(piextGetMemoryConnection)

_PI_API(piextKernelSetArgMemObj)
_PI_API(piextKernelSetArgSampler)

Expand Down
35 changes: 31 additions & 4 deletions sycl/include/sycl/detail/pi.h
Original file line number Diff line number Diff line change
Expand Up @@ -1000,6 +1000,16 @@ typedef enum { PI_MEM_CONTEXT = 0x1106, PI_MEM_SIZE = 0x1102 } _pi_mem_info;

using pi_mem_info = _pi_mem_info;

// Represent different memory connections between devices, see
// piextGetMemoryConnection.
typedef enum {
PI_MEMORY_CONNECTION_NONE, // copies must go through host
PI_MEMORY_CONNECTION_MIGRATABLE, // copies must be explicit
PI_MEMORY_CONNECTION_UNIFIED // copies are not needed
} _pi_memory_connection;

using pi_memory_connection = _pi_memory_connection;

//
// Following section contains SYCL RT Plugin Interface (PI) functions.
// They are 3 distinct categories:
Expand Down Expand Up @@ -1112,6 +1122,21 @@ __SYCL_EXPORT pi_result piextGetDeviceFunctionPointer(
pi_device device, pi_program program, const char *function_name,
pi_uint64 *function_pointer_ret);

/// Returns the type of memory connection between the two devices in the two
/// respective contexts.
///
/// \param dev1 First device.
/// \param ctx1 First context, must contain dev1.
/// \param dev2 Second device.
/// \param ctx2 Second context, must contain dev2.
/// \return res Type of memory connection supported between the pairs (dev1,
/// ctx1) and (dev2, ctx2).
__SYCL_EXPORT pi_result piextGetMemoryConnection(pi_device dev1,
pi_context ctx1,
pi_device dev2,
pi_context ctx2,
_pi_memory_connection *res);

//
// Context
//
Expand Down Expand Up @@ -1226,11 +1251,13 @@ __SYCL_EXPORT pi_result piextQueueCreateWithNativeHandle(
//
// Memory
//
__SYCL_EXPORT pi_result piMemBufferCreate(
pi_context context, pi_mem_flags flags, size_t size, void *host_ptr,
pi_mem *ret_mem, const pi_mem_properties *properties = nullptr);
__SYCL_EXPORT pi_result
piMemBufferCreate(pi_context context, pi_device device, pi_mem_flags flags,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did you need to add "device"? How is it used and what if it is not given?
We didn't need that for buffer migration in L0 BE.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the device, the memory will be allocated on. Even if we are putting multiple CUDA devices in a single context, they still have distinct device memories, so for the allocation we need to know what device to allocate on. It needs to be given and the runtime is changed in this PR so that it is always given. On backends that do not need this information device argument can be ignored.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The buffer would be represented by multiple allocations, one on each device in the context. It would then "migrate" (copy between allocations) depending on where the buffer is being accessed from. So what is that single "device" that you are adding and why it was not necessary for OpenCL and Level-Zero but is needed for CUDA?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For CUDA backend we want to be able to put multiple devices in the same context even when they do not share the same memory. So memory allocation can not be located in a context (that could contain different devices with distinct memories), but must be located on a specific device.

As far as I understand it other backends so far only supported putting multiple devices in the same context when they share memory.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, both OpenCL and Level-Zero backends support context with multiple devices in it even if they have their distinct memories. That's why I said there multiple allocations representing such a buffer in the context, and it is being copied essentially from device where it was used the last to where it is needed next.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that it is a possible way to go, but I am not sure it justifies the "complexity" of breaking backwards compatibility (btw, I am still not sure why you want to have some "initial" device for a buffer allocation since that can immediately change as app starts using it from other devices). The buffer "migration" across devices in the context implemented in OpenCL and Level-Zero is not a big deal, in my view, so how do we justify a re-course?

Adding few folks for extra opinions: @steffenlarsen , @romanovvlad

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(btw, I am still not sure why you want to have some "initial" device for a buffer allocation since that can immediately change as app starts using it from other devices)

There is no preemptive allocation on some device with this approach. You only have a device allocation when you are first using a buffer on that device. It can only migrate to another device after that allocation is first used on the device it was allocated on.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no preemptive allocation on some device with this approach. You only have a device allocation when you are first using a buffer on that device. It can only migrate to another device after that allocation is first used on the device it was allocated on.

So this is the device that first used that buffer, and nothing more? Eventually it will be migrated (by SYCL RT in your proposal) to whatever other device that needs it in this context, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exactly. Except if the backend returns PLUGIN_MANAGED_OR_SAME from piGetMemoryConnection. In this case RT will do nothing and assume the backend can handle any migrations it needs. That can be used for any pair of devices in the same context for the OpenCL backend.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added the query.

size_t size, void *host_ptr, pi_mem *ret_mem,
const pi_mem_properties *properties = nullptr);

__SYCL_EXPORT pi_result piMemImageCreate(pi_context context, pi_mem_flags flags,
__SYCL_EXPORT pi_result piMemImageCreate(pi_context context, pi_device device,
pi_mem_flags flags,
const pi_image_format *image_format,
const pi_image_desc *image_desc,
void *host_ptr, pi_mem *ret_mem);
Expand Down
Loading