From ac4089b10e1405a52726a9f5f4671cf7d925d00d Mon Sep 17 00:00:00 2001 From: KunwarSidhu47 Date: Fri, 14 Aug 2026 23:35:35 +0530 Subject: [PATCH 1/3] add node lock documentation Signed-off-by: KunwarSidhu47 --- docs/develop/protocol.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/develop/protocol.md b/docs/develop/protocol.md index 308ffd266c..8c1c61ea64 100644 --- a/docs/develop/protocol.md +++ b/docs/develop/protocol.md @@ -71,3 +71,12 @@ hami.io/vgpu-node: node67-4v100 hami.io/vgpu-time: 1705054796 ``` +## Node Lock Mechanism + +To prevent race conditions during concurrent pod scheduling, HAMi employs an annotation-based node locking mechanism (`hami.io/node-lock`). + +During the `Bind` phase, the scheduler acquires this lock on the target node before proceeding with device allocation. It is critical to understand the separation of concerns: +- **Annotation Lock:** The `hami.io/node-lock` annotation acts strictly as a concurrency mutex. +- **Device Accounting:** The actual accounting of GPU resources (memory, cores) is tracked independently. + +If a binding fails, releasing the lock (e.g., via a `fail()` fallback mechanism) **only removes the annotation lock**. It does not automatically revert or touch the underlying device accounting. Therefore, a prematurely released lock during an API error does not inherently cause hardware oversubscription, as the accounting state remains untouched. The device plugin is responsible for consuming this lock during pod creation to safely instantiate the required environment variables and mounts. From cf5b68caac433d5c78ac2bb738070931f5f93842 Mon Sep 17 00:00:00 2001 From: KunwarSidhu47 Date: Fri, 14 Aug 2026 23:49:02 +0530 Subject: [PATCH 2/3] fix annotation key name Signed-off-by: KunwarSidhu47 --- docs/develop/protocol.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/develop/protocol.md b/docs/develop/protocol.md index 8c1c61ea64..2f86230a13 100644 --- a/docs/develop/protocol.md +++ b/docs/develop/protocol.md @@ -73,10 +73,10 @@ hami.io/vgpu-time: 1705054796 ## Node Lock Mechanism -To prevent race conditions during concurrent pod scheduling, HAMi employs an annotation-based node locking mechanism (`hami.io/node-lock`). +To prevent race conditions during concurrent pod scheduling, HAMi employs an annotation-based node locking mechanism (`hami.io/mutex.lock`). During the `Bind` phase, the scheduler acquires this lock on the target node before proceeding with device allocation. It is critical to understand the separation of concerns: -- **Annotation Lock:** The `hami.io/node-lock` annotation acts strictly as a concurrency mutex. +- **Annotation Lock:** The `hami.io/mutex.lock` annotation acts strictly as a concurrency mutex. - **Device Accounting:** The actual accounting of GPU resources (memory, cores) is tracked independently. If a binding fails, releasing the lock (e.g., via a `fail()` fallback mechanism) **only removes the annotation lock**. It does not automatically revert or touch the underlying device accounting. Therefore, a prematurely released lock during an API error does not inherently cause hardware oversubscription, as the accounting state remains untouched. The device plugin is responsible for consuming this lock during pod creation to safely instantiate the required environment variables and mounts. From 79980abc7038e61a525bcee16230c2f412c14766 Mon Sep 17 00:00:00 2001 From: KunwarSidhu47 Date: Tue, 18 Aug 2026 16:41:01 +0530 Subject: [PATCH 3/3] address PR review comments Signed-off-by: KunwarSidhu47 --- docs/develop/protocol.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/develop/protocol.md b/docs/develop/protocol.md index 2f86230a13..8c7be4bf71 100644 --- a/docs/develop/protocol.md +++ b/docs/develop/protocol.md @@ -75,8 +75,10 @@ hami.io/vgpu-time: 1705054796 To prevent race conditions during concurrent pod scheduling, HAMi employs an annotation-based node locking mechanism (`hami.io/mutex.lock`). -During the `Bind` phase, the scheduler acquires this lock on the target node before proceeding with device allocation. It is critical to understand the separation of concerns: -- **Annotation Lock:** The `hami.io/mutex.lock` annotation acts strictly as a concurrency mutex. +Note that device allocation is actually decided in the `Filter` phase, not `Bind`. The lock is used to cover the critical window between the `Bind` phase and the device plugin's allocate phase. + +It is critical to understand the separation of concerns: +- **Annotation Lock:** The `hami.io/mutex.lock` annotation acts strictly as a concurrency mutex. The lock value is formatted as `,,` (e.g., `2024-01-23T04:30:00Z,default,my-pod`). By default, it expires after 5 minutes, though this can be overridden via the `HAMI_NODELOCK_EXPIRE` environment variable. The lock is released by the device plugin upon successful allocation, or by the scheduler if binding fails. - **Device Accounting:** The actual accounting of GPU resources (memory, cores) is tracked independently. -If a binding fails, releasing the lock (e.g., via a `fail()` fallback mechanism) **only removes the annotation lock**. It does not automatically revert or touch the underlying device accounting. Therefore, a prematurely released lock during an API error does not inherently cause hardware oversubscription, as the accounting state remains untouched. The device plugin is responsible for consuming this lock during pod creation to safely instantiate the required environment variables and mounts. +If a binding fails, releasing the lock **only removes the annotation lock**. It does not automatically revert or touch the underlying device accounting state. The device plugin is responsible for consuming this lock during pod creation to safely instantiate the required environment variables and mounts.