Skip to content
Merged
Show file tree
Hide file tree
Changes from 9 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions projects/amdsmi/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,14 @@ Full documentation for amd_smi_lib is available at [https://rocm.docs.amd.com/pr
- Identifies HBM Generation 4 VRAM, reported by `amdsmi_get_gpu_vram_info()`.
- Also added the pre-existing `HBM3E` value to the Python `AmdSmiVramType` enum, which had been missing it.

- **Added NPM (Node Power Management) balancing mode get/set**.
- New `amdsmi_npm_balancing_mode_t` enum: `AMDSMI_NPM_BALANCING_MODE_POWER_BALANCING` (PB, the default) or `AMDSMI_NPM_BALANCING_MODE_FREQUENCY_BALANCING` (FB).
- New APIs: `amdsmi_get_npm_balancing_mode()` and `amdsmi_set_npm_balancing_mode()`. Getting always reflects the last-selected mode regardless of NPM enablement, reporting `AMDSMI_NPM_BALANCING_MODE_INVALID` (not an error) only when the underlying value is missing or unreadable.
- Setting is AMD-SMI-only (not exposed via BMC Redfish/APML) and requires elevated privileges; it returns `AMDSMI_STATUS_NOT_SUPPORTED` without touching sysfs when NPM is disabled on the node.
- CLI: `amd-smi node -p` now shows `BALANCING_MODE`; new `amd-smi set --node-balancing-mode {POWER_BALANCING,FREQUENCY_BALANCING}` (node-wide, not per-GPU, requires sudo).
- New `AMDSMI_NPM_BALANCING_MODE_MAX` enum sentinel bounds the defined range, and new `amdsmi_get_npm_supported_balancing_modes()` reports the balancing modes supported by a node's platform/ASIC as a bitmask (bit N for enum value N), independent of NPM enablement. `amdsmi_set_npm_balancing_mode()` now also returns `AMDSMI_STATUS_NOT_SUPPORTED` when the requested mode is absent from this bitmask, reusing the same status as the NPM-disabled case (distinguishable by message text).
Comment thread
adam360x marked this conversation as resolved.
Resolved
- CLI: `amd-smi node -p` now also shows `SUPPORTED_BALANCING_MODES`; `amd-smi set --node-balancing-mode` reports a distinct "not supported on this platform" message for a mode absent from the bitmask vs. the generic "NPM is disabled" message.
Comment thread
adam360x marked this conversation as resolved.
Resolved

### Changed

- **`amdsmi_get_npm_info()` and `amdsmi_set_npm_limit()` now reject `amdsmi_node_handle` values not vended by `amdsmi_get_node_handle()`**.
Expand Down
100 changes: 100 additions & 0 deletions projects/amdsmi/amdsmi_cli/amdsmi_helpers.py
Original file line number Diff line number Diff line change
Expand Up @@ -3437,6 +3437,106 @@ def validate_and_set_power_cap(
}
return error_msg

# CLI-facing full names for the library's internal "PB"/"FB" strings
# (amdsmi_interface.py's own string contract stays unchanged).
NPM_BALANCING_MODE_TO_CLI = {"PB": "POWER_BALANCING", "FB": "FREQUENCY_BALANCING"}
NPM_BALANCING_MODE_FROM_CLI = {"POWER_BALANCING": "PB", "FREQUENCY_BALANCING": "FB"}

def validate_and_set_node_balancing_mode(self, node_handle, requested_mode, logger):
"""Validate and set the NPM balancing mode for a node.

Args:
node_handle: Node handle (system-wide, not per-GPU)
requested_mode (str): "POWER_BALANCING" or "FREQUENCY_BALANCING"
logger: AMDSMILogger instance for format-aware output

Returns:
dict or str: Structured data for JSON/CSV or formatted string for human-readable output
"""
if node_handle is None:
message = "No node handle available; cannot set balancing mode"
self.error_collector.record(int(amdsmi_cli_exceptions.AmdSmiExitCode.DEVICE_NOT_FOUND))
if logger.is_json_format() or logger.is_csv_format():
return {"status": "error", "message": message}
return message

# Pre-check NPM enablement, mirroring validate_and_set_node_power_limit():
# amdsmi_get_npm_supported_balancing_modes() is NOT gated on NPM
# enablement (it reflects hardware/firmware capability regardless of
# whether NPM is enabled), so a platform that supports both PB and FB
# would otherwise fall through to the generic "not supported on this
# node" message below even when the real cause is "NPM is disabled."
# Checking status up front lets that case be reported unambiguously.
try:
npm_info = amdsmi_interface.amdsmi_get_npm_info(node_handle)
if npm_info["status"] == amdsmi_interface.amdsmi_wrapper.AMDSMI_NPM_STATUS_DISABLED:
message = "[AMDSMI_STATUS_NOT_SUPPORTED] NPM is disabled on this node; cannot set balancing mode"
self.error_collector.record_library_error(
amdsmi_interface.amdsmi_wrapper.AMDSMI_STATUS_NOT_SUPPORTED
)
if logger.is_json_format() or logger.is_csv_format():
return {
"status": "error",
"requested_mode": requested_mode,
"error": "AMDSMI_STATUS_NOT_SUPPORTED",
"message": message,
}
return message
except amdsmi_exception.AmdSmiLibraryException:
# NPM enablement couldn't be determined (e.g. npm_status sysfs file
# unreadable); fall through and let the set call below report its
# own error rather than masking it with a pre-check failure.
pass

try:
amdsmi_interface.amdsmi_set_npm_balancing_mode(
node_handle, self.NPM_BALANCING_MODE_FROM_CLI[requested_mode]
)
message = f"Successfully set NPM balancing mode to {requested_mode}"
if logger.is_json_format() or logger.is_csv_format():
return {"status": "success", "balancing_mode": requested_mode, "message": message}
return message
except amdsmi_exception.AmdSmiLibraryException as e:
if e.get_error_code() == amdsmi_interface.amdsmi_wrapper.AMDSMI_STATUS_NO_PERM:
raise PermissionError("Command requires elevation") from e
if e.get_error_code() == amdsmi_interface.amdsmi_wrapper.AMDSMI_STATUS_NOT_SUPPORTED:
# NPM-disabled is already handled by the pre-check above, so a
# NOT_SUPPORTED reaching here means either "requested mode
# absent from this platform's supported_mode bitmask" or
# "board/mode sysfs file missing/unreadable while NPM is
# enabled" -- the status code alone can't distinguish those
# two. Query the supported-modes bitmask separately to tell
# the former apart from the latter, which stays lumped into
# the generic message since it's indistinguishable from here.
try:
supported = amdsmi_interface.amdsmi_get_npm_supported_balancing_modes(
node_handle
)
cli_supported_modes = [
self.NPM_BALANCING_MODE_TO_CLI.get(m, m) for m in supported
]
except amdsmi_exception.AmdSmiLibraryException:
cli_supported_modes = None

if cli_supported_modes is not None and requested_mode not in cli_supported_modes:
message = (
f"BALANCING_MODE: [{e.get_error_info(detailed=False)}] "
f"{requested_mode} is not supported on this platform"
)
else:
message = "NPM balancing mode is not supported on this node; cannot set balancing mode"
Comment thread
bill-shuzhou-liu marked this conversation as resolved.
Resolved
else:
message = f"[{e.get_error_info(detailed=False)}] Unable to set NPM balancing mode to {requested_mode}"
self.error_collector.record_library_error(e.get_error_code())
if logger.is_json_format() or logger.is_csv_format():
return {
"status": "error",
"requested_mode": requested_mode,
"error": e.get_error_info(detailed=False),
"message": message,
}
return message

def validate_and_set_node_power_limit(self, node_handle, requested_limit, logger):
"""Validate and set the NPM (Node Power Management) power limit for a node.

Expand Down
20 changes: 20 additions & 0 deletions projects/amdsmi/amdsmi_cli/amdsmi_parser.py
Original file line number Diff line number Diff line change
Expand Up @@ -2669,6 +2669,17 @@ def _add_set_value_parser(self, subparsers: argparse._SubParsersAction, func):
metavar="GB",
)

set_node_balancing_mode_help = "Set NPM balancing mode: POWER_BALANCING or FREQUENCY_BALANCING.\n\tThis is a system-wide setting, not per-GPU."
set_value_exclusive_group.add_argument(
"--node-balancing-mode",
action="store",
choices=["POWER_BALANCING", "FREQUENCY_BALANCING"],
type=str.upper,
required=False,
help=set_node_balancing_mode_help,
metavar="{POWER_BALANCING,FREQUENCY_BALANCING}",
)

# Node power limit is enabled on guest (1VF), maintain order
max_node_power_limit = self.helpers.get_max_node_power_limit()
set_node_power_limit_help = f"Set the node-level (NPM) power limit in watts.\n\tThis is a node-wide setting, not per-GPU.\n\tMax node power limit: {max_node_power_limit}"
Expand Down Expand Up @@ -2890,6 +2901,15 @@ def _add_set_value_parser(self, subparsers: argparse._SubParsersAction, func):

# Reject --gtt combined with --gpu at the argparse level
self._guard_gtt_gpu_conflict(set_value_parser, gtt_flags=("--gtt", "-G"))
# Improve the --gpu error message if combined with --node-balancing-mode;
# actual rejection happens at runtime in set_value.py, since these two
# flags sit in separate argparse groups and argparse itself never
# raises for this combination.
self._guard_gtt_gpu_conflict(
set_value_parser,
gtt_flags=("--node-balancing-mode",),
reason="--node-balancing-mode is a system-wide setting, not per-GPU",
)
# Improve the --gpu error message if combined with --node-power-limit;
# actual rejection happens at runtime in set_value.py, since these two
# flags sit in separate argparse groups and argparse itself never
Expand Down
39 changes: 39 additions & 0 deletions projects/amdsmi/amdsmi_cli/subcommands/node.py
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,9 @@ def node(
"limit": "N/A",
"status": "N/A",
"threshold": "N/A",
"balancing_mode": "N/A",
"current_node_power": "N/A",
"supported_balancing_modes": "N/A",
}
power_unit = "W"
limit = "N/A"
Expand Down Expand Up @@ -111,6 +113,30 @@ def node(
if current_node_power != "N/A":
npm_dict["current_node_power"] = current_node_power

# Balancing mode queries are not gated on NPM being enabled, so
# they run regardless of whether npm_info came back as a dict.
if args.nodes is not None:
try:
mode = amdsmi_interface.amdsmi_get_npm_balancing_mode(args.nodes)
npm_dict["balancing_mode"] = self.helpers.NPM_BALANCING_MODE_TO_CLI.get(
mode, mode
)
except amdsmi_exception.AmdSmiLibraryException as e:
logging.debug("amdsmi_get_npm_balancing_mode failed: %s", e.get_error_info())

try:
supported_modes = amdsmi_interface.amdsmi_get_npm_supported_balancing_modes(
args.nodes
)
npm_dict["supported_balancing_modes"] = [
self.helpers.NPM_BALANCING_MODE_TO_CLI.get(mode, mode)
for mode in supported_modes
]
except amdsmi_exception.AmdSmiLibraryException as e:
logging.debug(
"amdsmi_get_npm_supported_balancing_modes failed: %s", e.get_error_info()
)

# Get base board temperatures using node_handle
if args.base_board_temps:
if args.nodes is not None:
Expand Down Expand Up @@ -165,8 +191,16 @@ def node(
node_output.append(f" STATUS: {npm_dict.get('status', 'N/A')}")
threshold = npm_dict.get("threshold", "N/A")
node_output.append(f" THRESHOLD: {threshold} {power_unit}")
balancing_mode = npm_dict.get("balancing_mode", "N/A")
node_output.append(f" BALANCING_MODE: {balancing_mode}")
current_node_power = npm_dict.get("current_node_power", "N/A")
node_output.append(f" CURRENT_NODE_POWER: {current_node_power} {power_unit}")
supported_balancing_modes = npm_dict.get("supported_balancing_modes", "N/A")
if isinstance(supported_balancing_modes, list):
supported_balancing_modes = ", ".join(supported_balancing_modes)
node_output.append(
f" SUPPORTED_BALANCING_MODES: {supported_balancing_modes}"
)
if args.base_board_temps and base_board_temp_dict:
node_output.append(" BASEBOARD:")
node_output.append(" TEMPERATURE:")
Expand Down Expand Up @@ -199,7 +233,12 @@ def node(
csv_dict["limit"] = npm_dict.get("limit", "N/A")
csv_dict["status"] = npm_dict.get("status", "N/A")
csv_dict["threshold"] = npm_dict.get("threshold", "N/A")
csv_dict["balancing_mode"] = npm_dict.get("balancing_mode", "N/A")
csv_dict["current_node_power"] = npm_dict.get("current_node_power", "N/A")
supported_balancing_modes = npm_dict.get("supported_balancing_modes", "N/A")
if isinstance(supported_balancing_modes, list):
supported_balancing_modes = ";".join(supported_balancing_modes)
csv_dict["supported_balancing_modes"] = supported_balancing_modes
if args.base_board_temps and base_board_temp_dict:
csv_dict.update(base_board_temp_dict)
if args.gtt and gtt_dict:
Expand Down
22 changes: 21 additions & 1 deletion projects/amdsmi/amdsmi_cli/subcommands/set_value.py
Original file line number Diff line number Diff line change
Expand Up @@ -2075,6 +2075,7 @@ def set_value(
core_msr_floor_limit=None,
mem_carveout=None,
gtt=None,
node_balancing_mode=None,
):
"""Issue reset commands to target gpu(s)

Expand Down Expand Up @@ -2120,7 +2121,7 @@ def set_value(
process_isolation (int, optional): Value override for args.process_isolation. Defaults to None.
Raises:
AmdSmiRequiredCommandException: If no device target or argument is provided
AmdSmiInvalidParameterException: If GPU/CPU/CORE arguments are combined or --gtt is misused
AmdSmiInvalidParameterException: If GPU/CPU/CORE arguments are combined or --gtt/--node-balancing-mode is misused
PermissionError: If a set operation requires elevation (AMDSMI_STATUS_NO_PERM)

Return:
Expand All @@ -2134,6 +2135,8 @@ def set_value(
args.cpu = cpu
if core:
args.core = core
if node_balancing_mode:
args.node_balancing_mode = node_balancing_mode

# Special GTT handling (system-wide, not per-GPU) — handle before device dispatch
if hasattr(args, "gtt") and args.gtt is not None:
Expand Down Expand Up @@ -2169,6 +2172,23 @@ def set_value(
self.logger.print_output()
return

# Special node balancing mode handling (system-wide, not per-GPU) — handle before device dispatch
if hasattr(args, "node_balancing_mode") and args.node_balancing_mode is not None:
if hasattr(args, "gpu") and args.gpu is not None:
msg = (
"amd-smi set: error: argument --node-balancing-mode: not allowed with argument "
"--gpu/-g (--node-balancing-mode is a system-wide setting, not per-GPU)"
)
raise AmdSmiInvalidParameterException(
"set", "--node-balancing-mode", self.helpers.get_output_format(), msg
)
result = self.helpers.validate_and_set_node_balancing_mode(
self.node_handle, args.node_balancing_mode, self.logger
)
self.logger.output["set_node_balancing_mode"] = result
self.logger.print_output()
return

# Special node power limit handling (node-wide, not per-GPU) — handle before device dispatch
if hasattr(args, "node_power_limit") and args.node_power_limit is not None:
if hasattr(args, "gpu") and args.gpu is not None:
Expand Down
17 changes: 17 additions & 0 deletions projects/amdsmi/docs/how-to/amdsmi-cli-tool.md
Original file line number Diff line number Diff line change
Expand Up @@ -668,6 +668,9 @@ Set Arguments:
-R, --process-isolation STATUS Enable or disable the GPU process isolation on a per partition basis: 0 for disable and 1 for enable.
--ptl-status STATUS Enable or disable the PTL on a GPU processor: 0 for disable and 1 for enable
--ptl-format FRMT1,FRMT2 Set the PTL format on a GPU processor. For example, --ptl-format I8,F32
--node-balancing-mode {POWER_BALANCING,FREQUENCY_BALANCING}
Set NPM balancing mode: POWER_BALANCING or FREQUENCY_BALANCING.
This is a system-wide setting, not per-GPU.
-n, --node-power-limit WATTS Set the node-level (NPM) power limit in watts.
This is a node-wide setting, not per-GPU.
Max node power limit: 6000 W
Expand Down Expand Up @@ -1072,6 +1075,20 @@ On systems without UALoE hardware/session, `amdsmi_get_tray_info()` returns
`AMDSMI_STATUS_NOT_SUPPORTED` and the `TRAY:` block (and the `tray`/
`max_acc_per_tray`/`tray_type` keys in `--json`/`--csv`) is omitted entirely.

`amd-smi node --power-management` includes `BALANCING_MODE`, the NPM
balancing mode (`POWER_BALANCING`, the default, or `FREQUENCY_BALANCING`).
This field reads `N/A` when the underlying value is missing or unreadable,
not when NPM is disabled (see `STATUS` for enablement). Use
`amd-smi set --node-balancing-mode {POWER_BALANCING,FREQUENCY_BALANCING}` to
change it (AMD-SMI-only; not exposed via BMC Redfish/APML).

It also includes `SUPPORTED_BALANCING_MODES`, the balancing modes supported by
this node's platform/ASIC (e.g. `POWER_BALANCING, FREQUENCY_BALANCING`),
independent of current NPM enablement. `amd-smi set --node-balancing-mode`
returns `AMDSMI_STATUS_NOT_SUPPORTED` with a message naming the requested mode
if it is absent from this set, distinct from the same status code's "NPM is
disabled on this node" message.

## Interpreting the output

When you run an `amd-smi` command, the tool presents detailed information
Expand Down
Loading
Loading