[SYCL][DOC] Update the SYCL Runtime Interface document with design details (#680)

Signed-off-by: Garima Gupta <[email protected]>

Author: garimagu

sycl/doc/PluginInterface.md

@@ -1,14 +1,13 @@
 # The DPC++ Runtime Plugin Interface.
 
-
 ## Overview
-The DPC++ Runtime Plugin Interface (PI) is the interface layer between
-device-agnostic part of the DPC++ runtime and the device-specific runtime layers
+The DPC++ Runtime Plugin Interface (PI) is an interface layer between the
+device-agnostic part of DPC++ runtime and the device-specific runtime layers
 which control execution on devices. It employs the “plugin” mechanism to bind
-to the device specific runtime layers similarly to what is used by libomptarget
+to the device specific runtime layers similar to what is used by libomptarget
 or OpenCL.
 
-The picture below illustrates the placement of the PI within the overall DPC++
+The picture below illustrates the placement of PI within the overall DPC++
 runtime stack. Dotted lines show components or paths which are not yet available
 in the runtime, but are likely to be developed.
 ![PI in DPC++ runtime architecture](images/RuntimeArchitecture.svg)
@@ -22,31 +21,85 @@ points implementing the PI interface. The DPC++ runtime collects those function
 pointers into a PI interface dispatch table - one per plugin - and uses this
 table to dispatch to the device(s) covered by the corresponding plugin.
 
-PI is based on a subset of OpenCL 1.2 runtime specification, it follows its
-platform, execution and memory models in all aspects except those explicitly
-mentioned in this document. A part of PI API types and functions have exact
+PI is based on a subset of OpenCL 1.2 runtime specification, it follows OpenCL's
+platform, execution and memory models in all aspects except for those explicitly
+mentioned in this document. Some of PI API types and functions have exact
 matches in OpenCL. Whenever there is such a match, the semantics also fully
-matches unless the differences are explicitly specified in this document. While
+match unless the differences are explicitly specified in this document. While
 PI has roots in OpenCL, it does have many differences, and the gap is likely
-to grow, for example in the areas of memory model and management, program
+to grow, for example in areas of memory model and management, program
 management.
 
 ## Discovery and linkage of PI implementations
 
 ![PI implementation discovery](images/PluginDiscovery.svg)
 
 Device discovery phase enumerates all available devices and their features by
-querying underlying plugins found in the system. This process is only performed
-once before any actual offload is attempted.
+querying underlying plugins found in the system. This process is performed when
+all attached platforms or devices are queried in an application; for example,
+during device selection.
 
 ### Plugin discovery
 
-Plugins are physically dynamic libraries stored somewhere in the system where
-the DPC++ runtime runs. TBD - design and describe the process in details.
+Plugins are physically dynamic libraries or shared objects.
+The process to discover plugins follows the following guidelines.
+
+The DPC++ Runtime reads the names of the plugins from a configuration file 
+at a predetermined location (TBD - Add this location). These plugins are
+searched at locations in env LD_LIBRARY_PATH on Linux and env PATH on Windows.
+(TBD - Extend to search the plugins at a path relative to the SYCL Runtime
+installation directory by using DT_RPATH on Linux. Similar functionality can be
+achieved on Windows using SetDllDirectory. This will help avoiding extra setting
+of LD_LIBRARY_PATH.)
+To avoid any issues with read-only access, an environment variable SYCL_PI_CONFIG
+can be set to point to the configuration file which lists the Plugin names. The
+enviroment variable if set overrides the predetermined location's config file.
+These Plugins are then be searched in LD_LIBRARY_PATH locations.
+It is the developer's responsibility to include the plugin names from the
+predetermined location's config file to enable discovery of all plugins.
+(TBD - Extend to support search in DT_RPATH as above.)
+In the current implementation the plugin names are hardcoded in the library.
+Configuration file or env SYCL_PI_CONFIG is currently not being considered.
+
+A trace mechanism is provided using env SYCL_PI_TRACE to log the discovery/
+binding/ device enumeration process. Different levels of tracing can be achieved
+with different values of SYCL_PI_TRACE.
+SYCL_PI_TRACE=0x01 provides basic trace of plugins discovered and bound. It also
+lists the device selector's selected device information.
+SYCL_PI_TRACE=0x02 provides trace of all PI calls made from the DPC++ runtime
+with arguments and returned values.
+SYCL_PI_TRACE=-1 lists all PI Traces above and more debug messages.
 
 #### Plugin binary interface
-TBD - list and describe all the symbols plugin must export in order to be picked
-up by the DPC++ runtime for offload.
+Plugins should implement all the Interface APIs required for the PI Version
+it supports. There is [pi.def](../include/CL/sycl/detail/pi.def)/ 
+[pi.h](../include/CL/sycl/detail/pi.h) file listing all PI API names that can be
+called by the specific version of Plugin Interface.
+It exports a function - "piPluginInit" that returns the plugin details and function pointer
+table containing the list of pointers to implemented Interface Functions defined in pi.h.
+In the future, this document will list the minimum set of Interface APIs
+to be supported by Plugins. This will also require adding functionality to SYCL
+Runtime to work with such limited functionality plugins.
+
+(TBD - list and describe the symbols that a plugin must implement in order to
+be picked up by the DPC++ runtime for offload.)
+
+#### Binding a Plugin
+The DPC++ Runtime loads all discovered Plugins and tries to bind them by calling
+piPluginInit API for each loaded Plugin. The Plugins return the information of 
+supported PI version and the list of implemented PI API Function pointers.
+(TBD - Use the PI API Version information and check for compatibility.
+Extend to support version compatibility checks without loading the library.
+Eg:Changing the plugin name to reflect the supported Plugin Interface version.)
+The information of compatible plugins (with the Function Pointer Table) is
+stored in the associated platforms during platform object construction.
+The PI API calls are later forwarded using this information.
+A plugin is said to "bind" after this process completes with no errors.
+During device selection, the user can prefer selection of a device from a
+specific Plugin or Backend using the env SYCL_BE. The correspondence between
+a plugin and a SYCL_BE value is currently hardcoded in the runtime. 
+( TBD: Make this a part of configuration file). 
+Eg: SYCL_BE=PI_OPENCL corresponds to OpenCL Plugin.
 
 #### OpenCL plugin
 
@@ -56,12 +109,18 @@ OpenCL implementations. They can be installed either in the standard Khronos
 ICD-compatible way (e.g. listed in files under /etc/OpenCL/vendors on
 Linux) or not, and the OpenCL plugin can hook up with both.
 
-TBD describe the nested OpenCL implementation discovery process performed by
+TBD - implement and describe the nested OpenCL implementation discovery process performed by
 the OpenCL plugin
 
 ### Device enumeration by plugins
+Devices from all bound plugins are queried and listed as and when required, eg: during device selection in device_selector.
+The trace shows the PI API calls made when using SYCL_PI_TRACE=-1.
+(TBD - Add the trace to list all available devices when plugins are successfully bound.)
+
+### Plugin Unloading
+The plugins not chosen to be connected to should be unloaded.
 
-TBD
+TBD - Unloading a bound plugin.
 
 ## PI API Specification
 
@@ -71,11 +130,11 @@ able to operate on the corresponding device. The core API further breaks down
 into
   - **OpenCL-based** APIs which have OpenCL origin and semantics
   - **Extension** APIs which don't have counterparts in the OpenCL
-- **Interoperability API** which allows interoperability with underlying APIs
+- **Interoperability API** which allows interoperability with underlying runtimes
 such as OpenCL.
 
 See [pi.h](../include/CL/sycl/detail/pi.h) header for the full list and
-descriptions of PI APIs. [TBD: link to pi.h doxygen here]
+descriptions of PI APIs.
 
 ### The Core OpenCL-based PI APIs
 
@@ -114,10 +173,10 @@ in a data section.
 ### The Interoperability PI APIs
 
 These are APIs needed to implement DPC++ runtime interoperability with
-underlying "native" device runtimes such as OpenCL. Currently there are only
-OpenCL interoperability APIs, which is to be implemented by the OpenCL PI
-plugin only.  These APIs match semantics of the corresponding OpenCL APIs
-exactly.
+underlying "native" device runtimes such as OpenCL.
+There are some OpenCL interoperability APIs, which are to be implemented
+by the OpenCL PI plugin only. These APIs match semantics of the corresponding
+OpenCL APIs exactly.
 For example:
 
 ```
@@ -129,6 +188,21 @@ pi_result piclProgramCreateWithSource(
   pi_program *      ret_program);
 ```
 
+Some interoperability extension APIs have been added to get native runtime
+handles from the backend-agnostic PI Objects or to create PI Objects using the
+native handles. Eg:
+
+```
+pi_result piextDeviceGetNativeHandle(
+  pi_device device,
+  pi_native_handle *nativeHandle);
+
+pi_result piextDeviceCreateWithNativeHandle(
+  pi_native_handle nativeHandle,
+  pi_device *device);
+
+```
+
 ### PI Extension mechanism
 
 TBD This section describes a mechanism for DPC++ or other runtimes to detect