From 38e94ee826775f00796b4f95477188b3dde68585 Mon Sep 17 00:00:00 2001 From: Jiaying Zhang Date: Thu, 31 Aug 2017 11:33:13 -0700 Subject: [PATCH 1/4] Add device plugin doc under concepts/cluster-administration. --- _data/concepts.yml | 1 + .../cluster-administration/device-plugins.md | 88 +++++++++++++++++++ 2 files changed, 89 insertions(+) create mode 100644 docs/concepts/cluster-administration/device-plugins.md diff --git a/_data/concepts.yml b/_data/concepts.yml index 6740057bcfbf4..7550a51d2c373 100644 --- a/_data/concepts.yml +++ b/_data/concepts.yml @@ -90,6 +90,7 @@ toc: - docs/concepts/cluster-administration/master-node-communication.md - docs/concepts/cluster-administration/proxies.md - docs/concepts/cluster-administration/controller-metrics.md + - docs/concepts/cluster-administration/device-plugins.md - title: Policies section: - docs/concepts/policy/resource-quotas.md diff --git a/docs/concepts/cluster-administration/device-plugins.md b/docs/concepts/cluster-administration/device-plugins.md new file mode 100644 index 0000000000000..dc38f1ddbbccd --- /dev/null +++ b/docs/concepts/cluster-administration/device-plugins.md @@ -0,0 +1,88 @@ +--- +approvers: +title: Device Plugins +--- + +* TOC +{:toc} + +__Disclaimer__: Device plugins are in alpha. Its contents may change rapidly. + +Starting from 1.8 release, Kubernetes provides a [device plugin framework](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/device-plugin.md) +for vendors to advertise their resources to Kubelet without changing Kubernetes core code. +Instead of writing custom Kubernetes code, vendors can implement a device plugin that can +be deployed as a DaemonSet or in bare metal mode. The targeted devices include GPUs, +High-performance NICs, FPGAs, InfiniBand, and other similar computing resources +that may require vendor specific initialization and setup. + +## Overview +The 1.8 Kubernetes release supports device plugin as an alpha feature that +is gated by DevicePlugins feature gate and is disabled by default. +When the DevicePlugins feature is enabled, Kubelet will export a `Registration` gRPC service: +```gRPC +service Registration { + rpc Register(RegisterRequest) returns (Empty) {} +} +``` +A device plugin can register itself to Kubelet through this gRPC service. +During the registration, the device plugin needs to send + * The name of their unix socket + * The API version against which they were built + * The `ResourceName` they want to advertise. Here `ResourceName` needs to follow + the [extended resource naming scheme](https://github.com/kubernetes/kubernetes/pull/48922) as `vendor-domain/resource`. + E.g., Nvidia GPUs are advertised as `nvidia.com/gpu` + +Following a successful registration, the device plugin will send Kubelet the +list of devices it manages, and Kubelet will be in charge of advertising those +resources to the API server as part of the Kubelet node status update. +E.g., after a device plugin registers `vendor-domain/foo` with Kubelet +and reports two healthy devices on a node, the node status will be updated +to advertise 2 `vendor-domain/foo`. +Devices can then be selected using the same process as for OIRs in the pod spec. +Currently, extended resources are only spported as integer resources and expect +to always have limits == requests in container resource Spec. + +## Device Plugin Implementation + +The general workflow of a device plugin includes the following steps: +* Initialization. During this phase, the device plugin performs vendor specific initialization and setup to make sure the devices are in ready state. +* The plugin starts a gRPC service with a unix socket under host path: `/var/lib/kubelet/device-plugins/` that implements the following interfaces: +```gRPC +service DevicePlugin { + // ListAndWatch returns a stream of List of Devices + // Whenever a Device state change or a Device disapears, ListAndWatch + // returns the new list + rpc ListAndWatch(Empty) returns (stream ListAndWatchResponse) {} + + // Allocate is called during container creation so that the Device + // Plugin can run device specific operations and instruct Kubelet + // of the steps to make the Device available in the container + rpc Allocate(AllocateRequest) returns (AllocateResponse) {} +} +``` +* The plugin registers itself with Kubelet through unix socket at host path `/var/lib/kubelet/device-plugins/kubelet.sock`. +* After successfully registering itself, the device plugin runs in serving mode during which it keeps +monitoring device health and reports back to Kubelet upon any device state changes. +It is also responsible for serving Allocate gRPC requests. During Allocate, the device plugin may +perform device specific preparing operations (e.g., gpu cleanup, QRNG initialization, and etc.). +If the operations succeed, the device plugin will return an AllocateResponse that contains container +runtime configurations for accessing the allocated devices that Kubelet will pass to container runtime. + +A device plugin is expected to detect Kubelet restarts and re-register itself with the new +Kubelet instance. Currently, a new Kubelet instance will clean up all the existing unix sockets +under `/var/lib/kubelet/device-plugins` when it starts. A device plugin can monitor the deletion +of its unix socket and re-register itself upon such event. + +## Device Plugin Deployment + +A device plugin can be deployed as a DaemonSet or in bare metal mode. Being deployed as a DaemonSet has +the benefit that Kubernetes can restart the device plugins when they fail. +Otherwise, extra mechanism is needed to recover from device plugin failures. +The canonical directory `/var/lib/kubelet/device-plugins` requires priveledge access +so device plugin needs to run in privileged security context. It also needs to be mounted +as a volume in device plugin pod spec when running as a DaemonSet. + +## Examples + +For an example device plugin implementation, please check +[nvidia GPU device plugin for COS base OS](https://github.com/GoogleCloudPlatform/container-engine-accelerators/tree/master/nvidia_gpu). From f509e166f8ddd380c182ff42049b9368d609bc35 Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Thu, 7 Sep 2017 13:32:25 -0700 Subject: [PATCH 2/4] Update device-plugins.md --- .../cluster-administration/device-plugins.md | 127 +++++++++++------- 1 file changed, 76 insertions(+), 51 deletions(-) diff --git a/docs/concepts/cluster-administration/device-plugins.md b/docs/concepts/cluster-administration/device-plugins.md index dc38f1ddbbccd..8ce4fe0346c7b 100644 --- a/docs/concepts/cluster-administration/device-plugins.md +++ b/docs/concepts/cluster-administration/device-plugins.md @@ -3,52 +3,65 @@ approvers: title: Device Plugins --- -* TOC -{:toc} +{% include feature-state-alpha.md %} -__Disclaimer__: Device plugins are in alpha. Its contents may change rapidly. - -Starting from 1.8 release, Kubernetes provides a [device plugin framework](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/device-plugin.md) -for vendors to advertise their resources to Kubelet without changing Kubernetes core code. +{% capture overview %} +Starting in version 1.8, Kubernetes provides a [device plugin framework](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/device-plugin.md) +for vendors to advertise their resources to the kubelet without changing Kubernetes core code. Instead of writing custom Kubernetes code, vendors can implement a device plugin that can be deployed as a DaemonSet or in bare metal mode. The targeted devices include GPUs, High-performance NICs, FPGAs, InfiniBand, and other similar computing resources that may require vendor specific initialization and setup. +{% endcapture %} + +{% capture body %} + +## Device plugin registration + +The device plugins feature is gated by the `DevicePlugins` feature gate and is disabled by default. +When the device plugins feature is enabled, the kubelet exports a `Registration` gRPC service: -## Overview -The 1.8 Kubernetes release supports device plugin as an alpha feature that -is gated by DevicePlugins feature gate and is disabled by default. -When the DevicePlugins feature is enabled, Kubelet will export a `Registration` gRPC service: ```gRPC service Registration { rpc Register(RegisterRequest) returns (Empty) {} } ``` -A device plugin can register itself to Kubelet through this gRPC service. -During the registration, the device plugin needs to send - * The name of their unix socket - * The API version against which they were built - * The `ResourceName` they want to advertise. Here `ResourceName` needs to follow - the [extended resource naming scheme](https://github.com/kubernetes/kubernetes/pull/48922) as `vendor-domain/resource`. - E.g., Nvidia GPUs are advertised as `nvidia.com/gpu` - -Following a successful registration, the device plugin will send Kubelet the -list of devices it manages, and Kubelet will be in charge of advertising those -resources to the API server as part of the Kubelet node status update. -E.g., after a device plugin registers `vendor-domain/foo` with Kubelet -and reports two healthy devices on a node, the node status will be updated +A device plugin can register itself with the kubelet through this gRPC service. +During the registration, the device plugin needs to send: + + * The name of its Unix socket. + * The API version against which it was built. + * The `ResourceName` it wants to advertise. Here `ResourceName` needs to follow the + [extended resource naming scheme](https://github.com/kubernetes/kubernetes/pull/48922) + as `vendor-domain/resource`. + For example, an Nvidia GPU is advertised as `nvidia.com/gpu`. + +Following a successful registration, the device plugin sends the kubelet the +list of devices it manages, and the kubelet is then in charge of advertising those +resources to the API server as part of the kubelet node status update. +For example, after a device plugin registers `vendor-domain/foo` with the kubelet +and reports two healthy devices on a node, the node status is updated to advertise 2 `vendor-domain/foo`. -Devices can then be selected using the same process as for OIRs in the pod spec. -Currently, extended resources are only spported as integer resources and expect -to always have limits == requests in container resource Spec. -## Device Plugin Implementation +Then, developers can request devices in a +[Container](/docs/api-reference/{{page.version}}/#container-v1-core) +specification by using the same process that is used for +[opaque integer resources](/docs/tasks/configure-pod-container/opaque-integer-resource/). +In version 1.8, extended resources are spported only as integer resources and must have +`limit` equal to `request` in the Container specification. + +## Device plugin implementation The general workflow of a device plugin includes the following steps: -* Initialization. During this phase, the device plugin performs vendor specific initialization and setup to make sure the devices are in ready state. -* The plugin starts a gRPC service with a unix socket under host path: `/var/lib/kubelet/device-plugins/` that implements the following interfaces: -```gRPC -service DevicePlugin { + +* Initialization. During this phase, the device plugin performs vendor specific + initialization and setup to make sure the devices are in a ready state. + +* The plugin starts a gRPC service, with a Unix socket under host path + `/var/lib/kubelet/device-plugins/`, that implements the following interfaces: + + ```gRPC + service DevicePlugin { // ListAndWatch returns a stream of List of Devices // Whenever a Device state change or a Device disapears, ListAndWatch // returns the new list @@ -58,31 +71,43 @@ service DevicePlugin { // Plugin can run device specific operations and instruct Kubelet // of the steps to make the Device available in the container rpc Allocate(AllocateRequest) returns (AllocateResponse) {} -} -``` -* The plugin registers itself with Kubelet through unix socket at host path `/var/lib/kubelet/device-plugins/kubelet.sock`. -* After successfully registering itself, the device plugin runs in serving mode during which it keeps -monitoring device health and reports back to Kubelet upon any device state changes. -It is also responsible for serving Allocate gRPC requests. During Allocate, the device plugin may -perform device specific preparing operations (e.g., gpu cleanup, QRNG initialization, and etc.). -If the operations succeed, the device plugin will return an AllocateResponse that contains container -runtime configurations for accessing the allocated devices that Kubelet will pass to container runtime. - -A device plugin is expected to detect Kubelet restarts and re-register itself with the new -Kubelet instance. Currently, a new Kubelet instance will clean up all the existing unix sockets + } + ``` + +* The plugin registers itself with the kubelet through the Unix socket at host + path `/var/lib/kubelet/device-plugins/kubelet.sock`. + +* After successfully registering itself, the device plugin runs in serving mode, during which it keeps +monitoring device health and reports back to the kubelet upon any device state changes. +It is also responsible for serving `Allocate` gRPC requests. During `Allocate`, the device plugin may +do device-specific preparation; for example, GPU cleanup or QRNG initialization. +If the operations succeed, the device plugin returns an `AllocateResponse` that contains container +runtime configurations for accessing the allocated devices. The kubelet passes this information +to the container runtime. + +A device plugin is expected to detect kubelet restarts and re-register itself with the new +kubelet instance. In version 1.8, a new kubelet instance cleans up all the existing Unix sockets under `/var/lib/kubelet/device-plugins` when it starts. A device plugin can monitor the deletion -of its unix socket and re-register itself upon such event. +of its Unix socket and re-register itself upon such an event. -## Device Plugin Deployment +## Device plugin deployment A device plugin can be deployed as a DaemonSet or in bare metal mode. Being deployed as a DaemonSet has -the benefit that Kubernetes can restart the device plugins when they fail. -Otherwise, extra mechanism is needed to recover from device plugin failures. -The canonical directory `/var/lib/kubelet/device-plugins` requires priveledge access -so device plugin needs to run in privileged security context. It also needs to be mounted -as a volume in device plugin pod spec when running as a DaemonSet. +the benefit that Kubernetes can restart the device plugin if it fails. +Otherwise, an extra mechanism is needed to recover from device plugin failures. +The canonical directory `/var/lib/kubelet/device-plugins` requires priveledge access, +so a device plugin must run in a privileged security context. +If a device plugin is running as a DaemonSet, `/var/lib/kubelet/device-plugins` +must be mounted as a +[Volume](/docs/api-reference/{{page.version}}/#volume-v1-core) +in the plugin's +[PodSpec](/docs/api-reference/{{paage.version}}/#podspec-v1-core). ## Examples -For an example device plugin implementation, please check +For an example device plugin implementation, see [nvidia GPU device plugin for COS base OS](https://github.com/GoogleCloudPlatform/container-engine-accelerators/tree/master/nvidia_gpu). + +{% endcapture %} + +{% include templates/concept.md %} From cedc291ef86f1319acb07ddd62c64be1864ca780 Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Thu, 7 Sep 2017 14:39:23 -0700 Subject: [PATCH 3/4] Update device-plugins.md Add meta description. Fix typo. Change bare metal deployment to manual deployment. --- docs/concepts/cluster-administration/device-plugins.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/concepts/cluster-administration/device-plugins.md b/docs/concepts/cluster-administration/device-plugins.md index 8ce4fe0346c7b..867b735343a91 100644 --- a/docs/concepts/cluster-administration/device-plugins.md +++ b/docs/concepts/cluster-administration/device-plugins.md @@ -1,6 +1,7 @@ --- approvers: title: Device Plugins +description: Use the Kubernetes device plugin framework to implement plugins for GPUs, NICs, FPGAs, InfiniBand, and similar resources that require vendor-specific setup. --- {% include feature-state-alpha.md %} @@ -9,7 +10,7 @@ title: Device Plugins Starting in version 1.8, Kubernetes provides a [device plugin framework](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/device-plugin.md) for vendors to advertise their resources to the kubelet without changing Kubernetes core code. Instead of writing custom Kubernetes code, vendors can implement a device plugin that can -be deployed as a DaemonSet or in bare metal mode. The targeted devices include GPUs, +be deployed manually or as a DaemonSet. The targeted devices include GPUs, High-performance NICs, FPGAs, InfiniBand, and other similar computing resources that may require vendor specific initialization and setup. {% endcapture %} @@ -92,10 +93,10 @@ of its Unix socket and re-register itself upon such an event. ## Device plugin deployment -A device plugin can be deployed as a DaemonSet or in bare metal mode. Being deployed as a DaemonSet has +A device plugin can be deployed manually or as a DaemonSet. Being deployed as a DaemonSet has the benefit that Kubernetes can restart the device plugin if it fails. Otherwise, an extra mechanism is needed to recover from device plugin failures. -The canonical directory `/var/lib/kubelet/device-plugins` requires priveledge access, +The canonical directory `/var/lib/kubelet/device-plugins` requires priveleged access, so a device plugin must run in a privileged security context. If a device plugin is running as a DaemonSet, `/var/lib/kubelet/device-plugins` must be mounted as a From 1561b1751104c7ca60b5602bc8a7ed5ae8878204 Mon Sep 17 00:00:00 2001 From: Steve Perry Date: Thu, 7 Sep 2017 14:41:55 -0700 Subject: [PATCH 4/4] Update device-plugins.md Fix typo again. --- docs/concepts/cluster-administration/device-plugins.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/concepts/cluster-administration/device-plugins.md b/docs/concepts/cluster-administration/device-plugins.md index 867b735343a91..de3176bd660a8 100644 --- a/docs/concepts/cluster-administration/device-plugins.md +++ b/docs/concepts/cluster-administration/device-plugins.md @@ -96,7 +96,7 @@ of its Unix socket and re-register itself upon such an event. A device plugin can be deployed manually or as a DaemonSet. Being deployed as a DaemonSet has the benefit that Kubernetes can restart the device plugin if it fails. Otherwise, an extra mechanism is needed to recover from device plugin failures. -The canonical directory `/var/lib/kubelet/device-plugins` requires priveleged access, +The canonical directory `/var/lib/kubelet/device-plugins` requires privileged access, so a device plugin must run in a privileged security context. If a device plugin is running as a DaemonSet, `/var/lib/kubelet/device-plugins` must be mounted as a