Update dynamic mig documents and examples

docs/config.md

@@ -45,20 +45,41 @@ helm install hami hami-charts/hami --set devicePlugin.deviceMemoryScaling=5 ...
 * `resourcePriority:`
   String type, vgpu task priority name, default: "nvidia.com/priority"
 
-# Container config envs
+# Pod configs: annotations
+
+* `nvidia.com/use-gpuuuid:` 
+  String type, ie: "GPU-AAA,GPU-BBB"
+  If set, devices allocated by this pod must be one of UUIDs defined in this string.
+* `nvidia.com/nouse-gpuuuid`
+  String type, ie: "GPU-AAA,GPU-BBB"
+  If set, devices allocated by this pod will NOT in UUIDs defined in this string.
+* `nvidia.com/nouse-gputype:`
+  String type, ie: "Tesla V100-PCIE-32GB, NVIDIA A10"
+  If set, devices allocated by this pod will NOT in types defined in this string.
+* `nvidia.com/use-gputype`
+  String type, ie: "Tesla V100-PCIE-32GB, NVIDIA A10"
+  If set, devices allocated by this pod MUST be one of types defined in this string.
+* `hami.io/node-scheduler-policy`
+  String type, "binpack" or "spread"
+  binpack: the scheduler will try to allocate the pod to used GPU nodes for execution. 
+  spread: the scheduler will try to allocate the pod to different GPU nodes for execution.
+* `hami.io/gpu-scheduler-policy`
+  String type, "binpack" or "spread"
+  binpack: the scheduler will try to allocate the pod to the same GPU card for execution.
+  spread:the scheduler will try to allocate the pod to different GPU card for execution. 
+* `nvidia.com/vgpu-mode`
+  String type, "hami-core" or "mig"
+  Which type of vgpu instance this pod wish to use
+
+
+# Container configs: env
 
 * `GPU_CORE_UTILIZATION_POLICY:`
   String type, "default", "force", "disable"
   default: "default"
   "default" means the dafault utilization policy
   "force" means the container will always limit the core utilization below "nvidia.com/gpucores"
   "disable" means the container will ignore the utilization limitation set by "nvidia.com/gpucores" during task execution
-
-* `ACTIVE_OOM_KILLER:`
-  Bool type, "true","false"
-  default: false
-  "true" means there will be a daemon process which monitors all running tasks inside this container, and instantly kill any process which exceeds the limitation set by "nvidia.com/gpumem" or "nvidia.com/gpumemory"
-
 * `CUDA_DISABLE_CONTROL`
   Bool type, "true","false"
   default: false

docs/config_cn.md

@@ -33,16 +33,39 @@ helm install vgpu vgpu-charts/vgpu --set devicePlugin.deviceMemoryScaling=5 ...
 * `resourcePriority:`
   字符串类型，表示申请任务的任务优先级，默认: "nvidia.com/priority"
 
+# Pod配置（在注解中指定）
+
+* `nvidia.com/use-gpuuuid:` 
+  字符串类型, 如: "GPU-AAA,GPU-BBB"
+  如果设置, 该任务申请的设备只能是字符串中定义的设备之一。
+* `nvidia.com/nouse-gpuuuid`
+  字符串类型, 如: "GPU-AAA,GPU-BBB"
+  如果设置, 该任务不能使用字符串中定义的任何设备
+* `nvidia.com/nouse-gputype:`
+  字符串类型, 如: "Tesla V100-PCIE-32GB, NVIDIA A10"
+  如果设置, 该任务不能使用字符串中定义的任何设备型号
+* `nvidia.com/use-gputype`
+  字符串类型, 如: "Tesla V100-PCIE-32GB, NVIDIA A10"
+  如果设置, 该任务申请的设备只能使用字符串中定义的设备型号。
+* `hami.io/gpu-scheduler-policy`
+  字符串类型, "binpack" 或 "spread"
+  spread:, 调度器会尽量将任务均匀地分配在不同GPU中
+  binpack: 调度器会尽量将任务分配在已分配的GPU中，从而减少碎片
+* `hami.io/node-scheduler-policy`
+  字符串类型, "binpack" 或 "spread"
+  spread: 调度器会尽量将任务均匀地分配到不同节点上
+  binpack: 调度器会尽量将任务分配在已分配任务的节点上，从而减少碎片 
+* `nvidia.com/vgpu-mode`
+  字符串类型, "hami-core" 或 "mig"
+  该任务希望使用的vgpu类型
+
+
 # 容器配置（在容器的环境变量中指定）
 
 * `GPU_CORE_UTILIZATION_POLICY:`
   字符串类型，"default", "force", "disable"
   默认为"default"
   代表容器算力限制策略， "default"为默认，"force"为强制限制算力，一般用于测试算力限制的功能，"disable"为忽略算力限制
-* `ACTIVE_OOM_KILLER:`
-  布尔类型，"true", "false"
-  默认为false
-  若设置为true，则代表监控系统将会持续监控进程的显存使用量，并主动kill掉任何用超配额的进行。
 * `CUDA_DISABLE_CONTROL`
   布尔类型，"true", "false"
   默认为false

docs/dynamic-mig-support.md

@@ -0,0 +1,164 @@
+## Introduction
+
+**We now support dynamic-mig by using mig-parted to adjust mig-devices dynamically**, including:
+
+***Dynamic MIG instance management***: User don't need to operate on GPU node, using 'nvidia-smi -i 0 -mig 1' or other command to manage MIG instance, all will be done by HAMi-device-plugin.
+
+***Dynamic MIG Adjustment***: Each MIG device managed by HAMi will dyamically adjust their MIG template according to tasks submitted when necessary.
+
+***Device MIG Observation***: Each MIG instance generated by HAMi will be shown in scheduler-monitor, including task information. user can get a clear overview of MIG nodes.
+
+***Compatable with HAMi-core nodes***: HAMi can manage a unified GPU pool of `HAMi-core node` and `mig node`. A task can be scheduled to either node if not appointed manually by using `nvidia.com/vgpu-mode` annotation.
+
+***Unified API with HAMi-core***: Zero work needs to be done to make the job compatible with dynamic-mig feature.
+
+## Prerequisites
+
+* NVIDIA Blackwell and Hopper™ and Ampere Devices
+* HAMi > v2.5.0
+* Nvidia-container-toolkit
+
+## Enabling Dynamic-mig Support
+
+* Install the chart using helm, See 'enabling vGPU support in kubernetes' section [here](https://github.com/Project-HAMi/HAMi#enabling-vgpu-support-in-kubernetes)
+
+* Configure `mode` in device-plugin configMap to `mig` for MIG nodes
+```
+kubectl describe cm  hami-device-plugin -n kube-system
+```
+
+```json
+{
+    "nodeconfig": [
+        {
+            "name": "MIG-NODE-A",
+            "operatingmode": "mig",
+            "filterdevices": {
+              "uuid": [],
+              "index": []
+            }
+        }
+    ]
+}
+```
+
+* Restart the following pods for the change to take effect:
+  * hami-scheduler 
+  * hami-device-plugin on 'MIG-NODE-A'
+
+## Custom mig configuration (Optional)
+HAMi currently has a [built-in mig configuration](https://github.com/Project-HAMi/HAMi/blob/master/charts/hami/templates/scheduler/device-configmap.yaml) for MIG.
+
+You can customize the mig configuration by following the steps below:
+
+  ### Change the content of 'device-configmap.yaml' in charts/hami/templates/scheduler, the as follows
+
+  ```yaml
+    nvidia:
+      resourceCountName: {{ .Values.resourceName }}
+      resourceMemoryName: {{ .Values.resourceMem }}
+      resourceMemoryPercentageName: {{ .Values.resourceMemPercentage }}
+      resourceCoreName: {{ .Values.resourceCores }}
+      resourcePriorityName: {{ .Values.resourcePriority }}
+      overwriteEnv: false
+      defaultMemory: 0
+      defaultCores: 0
+      defaultGPUNum: 1
+      deviceSplitCount: {{ .Values.devicePlugin.deviceSplitCount }}
+      deviceMemoryScaling: {{ .Values.devicePlugin.deviceMemoryScaling }}
+      deviceCoreScaling: {{ .Values.devicePlugin.deviceCoreScaling }}
+      knownMigGeometries:
+      - models: [ "A30" ]
+        allowedGeometries:
+          - 
+            - name: 1g.6gb
+              memory: 6144
+              count: 4
+          - 
+            - name: 2g.12gb
+              memory: 12288
+              count: 2
+          - 
+            - name: 4g.24gb
+              memory: 24576
+              count: 1
+      - models: [ "A100-SXM4-40GB", "A100-40GB-PCIe", "A100-PCIE-40GB", "A100-SXM4-40GB" ]
+        allowedGeometries:
+          - 
+            - name: 1g.5gb
+              memory: 5120
+              count: 7
+          - 
+            - name: 2g.10gb
+              memory: 10240
+              count: 3
+            - name: 1g.5gb
+              memory: 5120
+              count: 1
+          - 
+            - name: 3g.20gb
+              memory: 20480
+              count: 2
+          - 
+            - name: 7g.40gb
+              memory: 40960
+              count: 1
+      - models: [ "A100-SXM4-80GB", "A100-80GB-PCIe", "A100-PCIE-80GB"]
+        allowedGeometries:
+          - 
+            - name: 1g.10gb
+              memory: 10240
+              count: 7
+          - 
+            - name: 2g.20gb
+              memory: 20480
+              count: 3
+            - name: 1g.10gb
+              memory: 10240
+              count: 1
+          - 
+            - name: 3g.40gb
+              memory: 40960
+              count: 2
+          - 
+            - name: 7g.79gb
+              memory: 80896
+              count: 1
+  ```
+
+  > **Note** Helm installation and updates will be based on the configuration in this file, overwriting the built-in configuration of Helm
+
+  > **Note** Be aware HAMi will find and use the first MIG template suitable to the task in the order of this configMap
+
+## Running MIG jobs
+
+MIG instance can now be requested by a container the same way as using `hami-core`
+simply by specifying the `nvidia.com/gpu` and `nvidia.com/gpumem` resource type.
+
+```yaml
+apiVersion: v1
+kind: Pod
+metadata:
+  name: gpu-pod
+  annotations:
+    nvidia.com/vgpu-mode: "mig" #(Optional), if not set, this pod can be assigned to a MIG instance or a hami-core instance
+spec:
+  containers:
+    - name: ubuntu-container
+      image: ubuntu:18.04
+      command: ["bash", "-c", "sleep 86400"]
+      resources:
+        limits:
+          nvidia.com/gpu: 2 
+          nvidia.com/gpumem: 8000
+```
+
+In this example above, the task allocates two mig instances, each with at least 8G device memory.
+
+## Notes
+
+1. You don't need to do anything on MIG node, all are managed by mig-parted in hami-device-plugin.
+
+2. Nvidia devices before Ampere architect can't use 'mig' mode
+
+3. You won't see any mig resources(ie, `nvidia.com/mig-1g.10gb`) on node, hami uses a unified resource name for both 'mig' and 'hami-core' node