Skip to content
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions docs/develop/protocol.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,3 +71,14 @@ hami.io/vgpu-node: node67-4v100
hami.io/vgpu-time: 1705054796
```

## Node Lock Mechanism

To prevent race conditions during concurrent pod scheduling, HAMi employs an annotation-based node locking mechanism (`hami.io/mutex.lock`).

Note that device allocation is actually decided in the `Filter` phase, not `Bind`. The lock is used to cover the critical window between the `Bind` phase and the device plugin's allocate phase.

It is critical to understand the separation of concerns:
- **Annotation Lock:** The `hami.io/mutex.lock` annotation acts strictly as a concurrency mutex. The lock value is formatted as `<time.RFC3339>,<podNamespace>,<podName>` (e.g., `2024-01-23T04:30:00Z,default,my-pod`). By default, it expires after 5 minutes, though this can be overridden via the `HAMI_NODELOCK_EXPIRE` environment variable. The lock is released by the device plugin upon successful allocation, or by the scheduler if binding fails.
- **Device Accounting:** The actual accounting of GPU resources (memory, cores) is tracked independently.

If a binding fails, releasing the lock **only removes the annotation lock**. It does not automatically revert or touch the underlying device accounting state. The device plugin is responsible for consuming this lock during pod creation to safely instantiate the required environment variables and mounts.
Loading