Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions projects/amdsmi/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ Full documentation for amd_smi_lib is available at [https://rocm.docs.amd.com/pr
- **Added NPM (Node Power Management) power limit setting**.
- New `amd-smi set --node-power-limit`/`-n` to set the node-level power limit in watts (node-wide, not per-GPU).
- New API: `amdsmi_set_npm_limit()`.
- New `current_node_power` field in `amdsmi_npm_info_t` (returned by `amdsmi_get_npm_info()`), the current (instantaneous) node power in watts, queried once per node rather than once per GPU; `amd-smi node -p` now displays it when available.
- New `current_node_power` field in `amdsmi_npm_info_t` (returned by `amdsmi_get_npm_info()`), the current (instantaneous) node power in watts. `amd-smi node -p` shows it as `CURRENT_NODE_POWER` (`N/A` when unavailable).
- New `max_node_power_limit` field in `amdsmi_npm_info_t` (returned by `amdsmi_get_npm_info()`), the platform max bound for `amdsmi_set_npm_limit()` requests.
- `amdsmi_set_npm_limit()` rejects the request with `AMDSMI_STATUS_INVAL` when NPM is disabled on the node (`amdsmi_npm_info_t::status == AMDSMI_NPM_STATUS_DISABLED`), instead of writing a limit that would have no defined effect. `amd-smi set --node-power-limit` performs the same check up front to fail fast with a friendlier message.

Expand Down Expand Up @@ -118,8 +118,11 @@ Full documentation for amd_smi_lib is available at [https://rocm.docs.amd.com/pr
- **`amdsmi_get_npm_info()` and `amdsmi_set_npm_limit()` now reject `amdsmi_node_handle` values not vended by `amdsmi_get_node_handle()`**.
- Previously any non-null handle was dereferenced directly; an unregistered/garbage handle now returns `AMDSMI_STATUS_INVAL` instead.

- **NPM sysfs numeric reads (e.g. `cur_node_power_limit`, `max_node_power_limit`) now reject negative or malformed content**.
- Previously a leading `-` (e.g. `"-1"`) parsed successfully as `UINT64_MAX`; such content now fails with `RSMI_STATUS_UNEXPECTED_DATA`.
- **`amd-smi node -p --csv` now starts with a `node` column**.
- Other `amd-smi node` CSV output (for example `--gtt` or `--base-board-temps` alone) has no `node` column.

- **`amd-smi set --gtt` no longer reports unrelated argument errors as a `--gpu` conflict**.
- Only `--gpu`/`-g` given without a value is rewritten to the `--gtt` conflict message; other argument errors are shown as-is.

- **`amd-smi`, `amd-smi version`, and `amd-smi static --driver` include the DKMS build in the amdgpu version**.
- For example `6.19.14.31400000-2370381`. Labels and JSON/CSV keys are unchanged.
Expand Down
77 changes: 12 additions & 65 deletions projects/amdsmi/amdsmi_cli/amdsmi_helpers.py
Original file line number Diff line number Diff line change
Expand Up @@ -1681,28 +1681,6 @@ def format_power_cap(self, value):
return f"{converted} W"
return value

def get_max_node_power_limit(self):
"""Get the platform max NPM node power limit (Watts), for help-text bounds.

Mirrors get_power_caps(): queried once at parser-build time so
--node-power-limit's help text can show its valid upper bound. NPM is
node-wide, so the first device that yields a reading is sufficient.
"""
for dev in self.get_gpu_handles():
try:
node_handle = amdsmi_interface.amdsmi_get_node_handle(dev)
max_node_power_limit = amdsmi_interface.amdsmi_get_npm_info(node_handle)[
"max_node_power_limit"
]
if max_node_power_limit != "N/A":
return f"{max_node_power_limit} W"
except (amdsmi_interface.AmdSmiLibraryException, KeyError) as e:
logging.debug(
f"AMDSMIHelpers.get_max_node_power_limit - Unable to get NPM info for device {dev}: {str(e)}"
)
continue
return "N/A"

@staticmethod
def detect_gpu_od(bdf):
"""Detect if a GPU has the gpu_od sysfs interface.
Expand Down Expand Up @@ -3629,34 +3607,7 @@ def validate_and_set_node_balancing_mode(self, node_handle, requested_mode, logg
return message

def validate_and_set_node_power_limit(self, node_handle, requested_limit, logger):
"""Validate and set the NPM (Node Power Management) power limit for a node.

Mirrors validate_and_set_power_cap(): fetches the platform max bound via
amdsmi_get_npm_info() (amdsmi_npm_info_t::max_node_power_limit, sourced from
board/max_node_power_limit) and rejects out-of-range requests before ever
issuing the write. This is an early, CLI-friendly pre-check only:
amdsmi_set_npm_limit() itself already validates `limit` against the same
bound internally and fails closed if that bound can't be read (see its
docstring); this CLI-side check exists purely to fail fast and surface a
clean, user-facing hint here rather than relying solely on the library
call's exception.

There is no driver-exposed minimum bound beyond positivity (no sysfs file
or design-doc source documents a real minimum), so the lower bound enforced
here is simply "> 0".

Also rejects the request up front if NPM itself is disabled on this node
(amdsmi_npm_info_t::status == AMDSMI_NPM_STATUS_DISABLED) -- there is no
max bound to validate against in that case, and writing
board/cur_node_power_limit while NPM is disabled has no defined effect.
Same fail-fast pattern as the max-bound check above: amdsmi_set_npm_limit()
itself already enforces this too, so this is a CLI-friendly early exit,
not the sole enforcement.

Fails closed when the platform max is unavailable ("N/A"): rather than
allowing any positive value through in that degraded-driver scenario, the
request is rejected outright here too, matching amdsmi_set_npm_limit()'s
own fail-closed behavior for the same condition.
"""Validate and set the NPM node power limit, rejecting before the write when out of range.

Args:
node_handle: Node handle obtained via amdsmi_get_node_handle()
Expand All @@ -3672,29 +3623,25 @@ def validate_and_set_node_power_limit(self, node_handle, requested_limit, logger

if npm_info["status"] == amdsmi_interface.amdsmi_wrapper.AMDSMI_NPM_STATUS_DISABLED:
raise amdsmi_cli_exceptions.AmdSmiInvalidParameterValueException(
sys.argv[1] if len(sys.argv) > 1 else "unknown",
"set",
f"{requested_limit}",
self.get_output_format(),
hint="Node power limit cannot be set: NPM is disabled on this node",
hint="--node-power-limit cannot be set: NPM is disabled on this node",
)
elif max_node_power_limit == "N/A":
# Fail closed: the platform max is unreadable (degraded driver /
# unavailable sysfs), so there is no bound to validate the
# request against. Reject rather than let an unbounded value
# through.
# Fail closed: without a platform max there is no bound to check against.
raise amdsmi_cli_exceptions.AmdSmiInvalidParameterValueException(
sys.argv[1] if len(sys.argv) > 1 else "unknown",
"set",
f"{requested_limit}",
self.get_output_format(),
hint="Node power limit cannot be validated: platform maximum is unavailable",
hint="--node-power-limit cannot be validated: platform maximum is unavailable",
)
elif not 0 < requested_limit <= max_node_power_limit:
# Raise so the caller exits with a non-zero return code
raise amdsmi_cli_exceptions.AmdSmiInvalidParameterValueException(
sys.argv[1] if len(sys.argv) > 1 else "unknown",
"set",
f"{requested_limit}",
self.get_output_format(),
hint=f"Node power limit must be between 1W and {max_node_power_limit}W",
hint=f"--node-power-limit must be between 1W and {max_node_power_limit}W",
)

amdsmi_interface.amdsmi_set_npm_limit(node_handle, requested_limit)
Expand All @@ -3709,15 +3656,15 @@ def validate_and_set_node_power_limit(self, node_handle, requested_limit, logger
if e.get_error_code() == amdsmi_interface.amdsmi_wrapper.AMDSMI_STATUS_NO_PERM:
raise PermissionError("Command requires elevation") from e
if e.get_error_code() == amdsmi_interface.amdsmi_wrapper.AMDSMI_STATUS_INVAL:
# Raise so the caller exits with a non-zero return code, same as
# the CLI-side bound checks above.
# Same exit code as the CLI-side range checks above.
raise amdsmi_cli_exceptions.AmdSmiInvalidParameterValueException(
sys.argv[1] if len(sys.argv) > 1 else "unknown",
"set",
f"{requested_limit}",
self.get_output_format(),
hint=e.get_error_info(detailed=False),
hint=f"--node-power-limit rejected: {e.get_error_info(detailed=False)}",
) from e
error_msg = f"[{e.get_error_info(detailed=False)}] Unable to set node power limit to {requested_limit} W"
self.error_collector.record_library_error(e.get_error_code())
if logger.is_json_format() or logger.is_csv_format():
return {
"status": "error",
Expand Down
12 changes: 1 addition & 11 deletions projects/amdsmi/amdsmi_cli/amdsmi_parser.py
Original file line number Diff line number Diff line change
Expand Up @@ -2690,8 +2690,7 @@ def _add_set_value_parser(self, subparsers: argparse._SubParsersAction, func):
)

# Node power limit is enabled on guest (1VF), maintain order
max_node_power_limit = self.helpers.get_max_node_power_limit()
set_node_power_limit_help = f"Set the node-level (NPM) power limit in watts.\n\tThis is a node-wide setting, not per-GPU.\n\tMax node power limit: {max_node_power_limit}"
set_node_power_limit_help = "Set the node-level (NPM) power limit in watts.\n\tThis is a node-wide setting, not per-GPU."
set_value_exclusive_group.add_argument(
"-n",
"--node-power-limit",
Expand Down Expand Up @@ -2919,15 +2918,6 @@ def _add_set_value_parser(self, subparsers: argparse._SubParsersAction, func):
gtt_flags=("--node-balancing-mode",),
reason="--node-balancing-mode is a system-wide setting, not per-GPU",
)
# Improve the --gpu error message if combined with --node-power-limit;
# actual rejection happens at runtime in set_value.py, since these two
# flags sit in separate argparse groups and argparse itself never
# raises for this combination.
self._guard_gtt_gpu_conflict(
set_value_parser,
gtt_flags=("--node-power-limit", "-n"),
reason="--node-power-limit is a node-wide setting, not per-GPU",
)

# Set accepts default devices of all
self._add_device_arguments(set_value_parser, required=False)
Expand Down
6 changes: 3 additions & 3 deletions projects/amdsmi/amdsmi_cli/subcommands/node.py
Original file line number Diff line number Diff line change
Expand Up @@ -226,10 +226,10 @@ def node(
print("\n".join(node_output))
else:
if self.logger.is_csv_format():
# Only node 0 is currently supported; mirror the gpu/cpu/nic
# CSV surfaces, which always lead with a device identifier.
csv_dict = {"node": 0}
csv_dict = {}
if args.power_management:
# Only node 0 exists today; the NPM design shows this column for -p output only.
csv_dict["node"] = 0
csv_dict["limit"] = npm_dict.get("limit", "N/A")
csv_dict["status"] = npm_dict.get("status", "N/A")
csv_dict["threshold"] = npm_dict.get("threshold", "N/A")
Expand Down
14 changes: 9 additions & 5 deletions projects/amdsmi/amdsmi_cli/subcommands/set_value.py
Original file line number Diff line number Diff line change
Expand Up @@ -2130,7 +2130,7 @@ def set_value(
process_isolation (int, optional): Value override for args.process_isolation. Defaults to None.
Raises:
AmdSmiRequiredCommandException: If no device target or argument is provided
AmdSmiInvalidParameterException: If GPU/CPU/CORE arguments are combined or --gtt/--node-balancing-mode is misused
AmdSmiInvalidParameterException: If GPU/CPU/CORE arguments are combined or --gtt/--node-balancing-mode/--node-power-limit is misused
PermissionError: If a set operation requires elevation (AMDSMI_STATUS_NO_PERM)

Return:
Expand Down Expand Up @@ -2201,18 +2201,22 @@ def set_value(
# Special node power limit handling (node-wide, not per-GPU) — handle before device dispatch
if hasattr(args, "node_power_limit") and args.node_power_limit is not None:
if hasattr(args, "gpu") and args.gpu is not None:
print(
msg = (
"amd-smi set: error: argument --node-power-limit/-n: not allowed with "
"argument --gpu/-g (--node-power-limit is a node-wide setting, not per-GPU)",
file=sys.stderr,
"argument --gpu/-g (--node-power-limit is a node-wide setting, not per-GPU)"
)
raise AmdSmiInvalidParameterException(
"set", "--node-power-limit", self.helpers.get_output_format(), msg
)
sys.exit(2)
requested_limit = args.node_power_limit
if self.node_handle is None:
self.logger.output["set_node_power_limit"] = (
"[AMDSMI_STATUS_NOT_SUPPORTED - Feature not supported] "
"Unable to set node power limit: no NPM-capable node found"
)
self.helpers.error_collector.record_library_error(
amdsmi_interface.amdsmi_wrapper.AMDSMI_STATUS_NOT_SUPPORTED
)
self.logger.print_output()
return
result = self.helpers.validate_and_set_node_power_limit(
Expand Down
1 change: 0 additions & 1 deletion projects/amdsmi/docs/how-to/amdsmi-cli-tool.md
Original file line number Diff line number Diff line change
Expand Up @@ -721,7 +721,6 @@ Set Arguments:
This is a system-wide setting, not per-GPU.
-n, --node-power-limit WATTS Set the node-level (NPM) power limit in watts.
This is a node-wide setting, not per-GPU.
Max node power limit: 6000 W

CPU Arguments:
--cpu-pwr-limit PWR_LIMIT Set power limit for the given socket. Input parameter is power limit value.
Expand Down
55 changes: 16 additions & 39 deletions projects/amdsmi/docs/reference/amdsmi-py-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -3102,43 +3102,19 @@ finally:

### amdsmi_set_npm_limit

Description: Set the NPM (Node Power Management) power limit for the node
associated with `node_handle` by writing to the board's
`cur_node_power_limit` sysfs interface.

This function rejects the request with `AmdSmiLibraryException`
(`AMDSMI_STATUS_INVAL`) if NPM is disabled on the node
(`amdsmi_get_npm_info()`'s `status` == `AMDSMI_NPM_STATUS_DISABLED`), since
writing `board/cur_node_power_limit` while NPM is disabled has no defined
effect. It also validates `limit` against the platform max bound
(`amdsmi_get_npm_info()`'s `max_node_power_limit`, sourced from
`board/max_node_power_limit`) internally before ever issuing the write,
raising `AmdSmiLibraryException` with `AMDSMI_STATUS_INVAL` if `limit` is `0`
or greater than that bound. If the platform max bound itself cannot be read
(e.g. the sysfs interface is missing or returns unexpected data), this
function fails closed and raises that underlying error rather than silently
allowing an unbounded `limit` through. The amd-smi CLI's
`set --node-power-limit` additionally performs the same checks itself ahead
of calling this function, purely to fail fast and present a friendlier,
earlier user-facing error message; it is not the only validation and is not
required for correctness.
Description: Sets the NPM (Node Power Management) power limit, in watts, for the
node associated with `node_handle`. Requires root. The library rejects the
request without writing if NPM is disabled or `limit` exceeds
`max_node_power_limit` from `amdsmi_get_npm_info`.

Input parameters:

* `node_handle` node handle obtained from `amdsmi_get_node_handle`
* `limit` new NPM power limit value to request (units match the
`board/cur_node_power_limit` sysfs interface). Must satisfy
`0 < limit <= UINT64_MAX`; out-of-range values raise
`AmdSmiParameterException` rather than silently wrapping modulo 2**64 the
way a raw `ctypes.c_uint64()` conversion would. Values that pass this local
bound check but are `0` or exceed the platform max are rejected by the
underlying library call instead (see Exceptions below)

Output: None. This function raises an exception if the call did not succeed
(e.g. `AmdSmiLibraryException` with `AMDSMI_STATUS_INVAL` if `limit` is `0` or
exceeds the platform max bound, `AMDSMI_STATUS_NOT_SUPPORTED` if the sysfs
interface is unavailable, or `AMDSMI_STATUS_NO_PERM` if the write was
rejected)
* `limit` new node power limit in watts. Must be an `int` (not `bool`) in
`1..2**64-1`; otherwise `AmdSmiParameterException` is raised before the
library is called (for example, for a `limit` of `0`)

Output: None

Exceptions that can be thrown by `amdsmi_set_npm_limit` function:

Expand All @@ -3147,12 +3123,13 @@ Exceptions that can be thrown by `amdsmi_set_npm_limit` function:

#### Possible Library Exceptions

- `AMDSMI_STATUS_INVAL` - Invalid parameters, including when `limit` is `0`
or exceeds the platform max bound (or when that bound itself cannot be
read)
- `AMDSMI_STATUS_NOT_SUPPORTED` - Feature not supported
- `AMDSMI_STATUS_NO_PERM` - Permission Denied (e.g. the write was rejected by
the driver for this guest context)
- `AMDSMI_STATUS_INVAL` - Invalid parameters, NPM is disabled, or `limit`
exceeds `max_node_power_limit`
- `AMDSMI_STATUS_NO_PERM` - Not root, or the driver rejected the write (for
example, from a guest that is not permitted to set it)
- `AMDSMI_STATUS_NOT_SUPPORTED` - NPM sysfs interface is missing
- `AMDSMI_STATUS_UNEXPECTED_DATA` - NPM sysfs content is unparsable or the
platform max is unavailable

Example:

Expand Down
Loading
Loading