Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions projects/amdsmi/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,13 @@ Full documentation for amd_smi_lib is available at [https://rocm.docs.amd.com/pr
- Identifies HBM Generation 4 VRAM, reported by `amdsmi_get_gpu_vram_info()`.
- Also added the pre-existing `HBM3E` value to the Python `AmdSmiVramType` enum, which had been missing it.

- **Added NPM (Node Power Management) balancing mode get/set**.
- New `amdsmi_npm_balancing_mode_t` enum: `AMDSMI_NPM_BALANCING_MODE_POWER_BALANCING` (PB, the default) or `AMDSMI_NPM_BALANCING_MODE_FREQUENCY_BALANCING` (FB).
- New APIs: `amdsmi_get_npm_balancing_mode()` and `amdsmi_set_npm_balancing_mode()`. Getting always reflects the last-selected mode regardless of NPM enablement, returning `AMDSMI_STATUS_NOT_SUPPORTED` only when the underlying value is missing or unreadable.
- Setting is AMD-SMI-only (not exposed via BMC Redfish/APML) and requires elevated privileges; it returns `AMDSMI_STATUS_NOT_SUPPORTED` without touching sysfs when NPM is disabled on the node.
- CLI: `amd-smi node -p` now shows `BALANCING_MODE`; new `amd-smi set --node-balancing-mode {POWER_BALANCING,FREQUENCY_BALANCING}` (node-wide, not per-GPU, requires sudo).
- New `AMDSMI_NPM_BALANCING_MODE_MAX` enum value marks the end of the defined range, and new `amdsmi_get_npm_supported_balancing_modes()` reports the balancing modes supported by a node's platform/ASIC as a bitmask (bit N for enum value N), independent of NPM enablement. `amdsmi_set_npm_balancing_mode()` now returns `AMDSMI_STATUS_SETTING_UNAVAILABLE` when the requested mode is absent from this bitmask, distinct from the `AMDSMI_STATUS_NOT_SUPPORTED` NPM-disabled case. CLI: `amd-smi node -p` now also shows `SUPPORTED_BALANCING_MODES`, and `amd-smi set --node-balancing-mode` reports a distinct "not supported on this platform" message for a mode absent from the bitmask vs. the generic "NPM is disabled" message.

### Changed

- **`amdsmi_get_npm_info()` and `amdsmi_set_npm_limit()` now reject `amdsmi_node_handle` values not vended by `amdsmi_get_node_handle()`**.
Expand Down
84 changes: 84 additions & 0 deletions projects/amdsmi/amdsmi_cli/amdsmi_helpers.py
Original file line number Diff line number Diff line change
Expand Up @@ -3437,6 +3437,90 @@ def validate_and_set_power_cap(
}
return error_msg

# CLI-facing full names for the library's internal "PB"/"FB" strings
# (amdsmi_interface.py's own string contract stays unchanged).
NPM_BALANCING_MODE_TO_CLI = {"PB": "POWER_BALANCING", "FB": "FREQUENCY_BALANCING"}
NPM_BALANCING_MODE_FROM_CLI = {"POWER_BALANCING": "PB", "FREQUENCY_BALANCING": "FB"}

def validate_and_set_node_balancing_mode(self, node_handle, requested_mode, logger):
"""Validate and set the NPM balancing mode for a node.

Args:
node_handle: Node handle (system-wide, not per-GPU)
requested_mode (str): "POWER_BALANCING" or "FREQUENCY_BALANCING"
logger: AMDSMILogger instance for format-aware output

Returns:
dict or str: Structured data for JSON/CSV or formatted string for human-readable output
"""
if node_handle is None:
message = "No node handle available; cannot set balancing mode"
self.error_collector.record(int(amdsmi_cli_exceptions.AmdSmiExitCode.DEVICE_NOT_FOUND))
if logger.is_json_format() or logger.is_csv_format():
return {"status": "error", "message": message}
return message

# Pre-check NPM enablement, mirroring validate_and_set_node_power_limit():
# amdsmi_set_npm_balancing_mode() reports both "NPM disabled" and
# "board/mode sysfs file missing/unreadable while NPM is enabled" as
# the same AMDSMI_STATUS_NOT_SUPPORTED. Checking status up front lets
# the former be reported unambiguously.
try:
npm_info = amdsmi_interface.amdsmi_get_npm_info(node_handle)
if npm_info["status"] == amdsmi_interface.amdsmi_wrapper.AMDSMI_NPM_STATUS_DISABLED:
message = "[AMDSMI_STATUS_NOT_SUPPORTED] NPM is disabled on this node; cannot set balancing mode"
self.error_collector.record_library_error(
amdsmi_interface.amdsmi_wrapper.AMDSMI_STATUS_NOT_SUPPORTED
)
if logger.is_json_format() or logger.is_csv_format():
return {
"status": "error",
"requested_mode": requested_mode,
"error": "AMDSMI_STATUS_NOT_SUPPORTED",
"message": message,
}
return message
except amdsmi_exception.AmdSmiLibraryException:
# NPM enablement couldn't be determined (e.g. npm_status sysfs file
# unreadable); fall through and let the set call below report its
# own error rather than masking it with a pre-check failure.
pass

try:
amdsmi_interface.amdsmi_set_npm_balancing_mode(
node_handle, self.NPM_BALANCING_MODE_FROM_CLI[requested_mode]
)
message = f"Successfully set NPM balancing mode to {requested_mode}"
if logger.is_json_format() or logger.is_csv_format():
return {"status": "success", "balancing_mode": requested_mode, "message": message}
return message
except amdsmi_exception.AmdSmiLibraryException as e:
if e.get_error_code() == amdsmi_interface.amdsmi_wrapper.AMDSMI_STATUS_NO_PERM:
raise PermissionError("Command requires elevation") from e
if (
e.get_error_code()
== amdsmi_interface.amdsmi_wrapper.AMDSMI_STATUS_SETTING_UNAVAILABLE
):
message = (
f"BALANCING_MODE: [{e.get_error_info(detailed=False)}] "
f"{requested_mode} is not supported on this platform"
)
elif e.get_error_code() == amdsmi_interface.amdsmi_wrapper.AMDSMI_STATUS_NOT_SUPPORTED:
message = (
"NPM balancing mode is not supported on this node; cannot set balancing mode"
)
else:
message = f"[{e.get_error_info(detailed=False)}] Unable to set NPM balancing mode to {requested_mode}"
self.error_collector.record_library_error(e.get_error_code())
if logger.is_json_format() or logger.is_csv_format():
return {
"status": "error",
"requested_mode": requested_mode,
"error": e.get_error_info(detailed=False),
"message": message,
}
return message

def validate_and_set_node_power_limit(self, node_handle, requested_limit, logger):
"""Validate and set the NPM (Node Power Management) power limit for a node.

Expand Down
20 changes: 20 additions & 0 deletions projects/amdsmi/amdsmi_cli/amdsmi_parser.py
Original file line number Diff line number Diff line change
Expand Up @@ -2669,6 +2669,17 @@ def _add_set_value_parser(self, subparsers: argparse._SubParsersAction, func):
metavar="GB",
)

set_node_balancing_mode_help = "Set NPM balancing mode: POWER_BALANCING or FREQUENCY_BALANCING.\n\tThis is a system-wide setting, not per-GPU."
set_value_exclusive_group.add_argument(
"--node-balancing-mode",
action="store",
choices=["POWER_BALANCING", "FREQUENCY_BALANCING"],
type=str.upper,
required=False,
help=set_node_balancing_mode_help,
metavar="{POWER_BALANCING,FREQUENCY_BALANCING}",
)

# Node power limit is enabled on guest (1VF), maintain order
max_node_power_limit = self.helpers.get_max_node_power_limit()
set_node_power_limit_help = f"Set the node-level (NPM) power limit in watts.\n\tThis is a node-wide setting, not per-GPU.\n\tMax node power limit: {max_node_power_limit}"
Expand Down Expand Up @@ -2890,6 +2901,15 @@ def _add_set_value_parser(self, subparsers: argparse._SubParsersAction, func):

# Reject --gtt combined with --gpu at the argparse level
self._guard_gtt_gpu_conflict(set_value_parser, gtt_flags=("--gtt", "-G"))
# Improve the --gpu error message if combined with --node-balancing-mode;
# actual rejection happens at runtime in set_value.py, since these two
# flags sit in separate argparse groups and argparse itself never
# raises for this combination.
self._guard_gtt_gpu_conflict(
set_value_parser,
gtt_flags=("--node-balancing-mode",),
reason="--node-balancing-mode is a system-wide setting, not per-GPU",
)
# Improve the --gpu error message if combined with --node-power-limit;
# actual rejection happens at runtime in set_value.py, since these two
# flags sit in separate argparse groups and argparse itself never
Expand Down
39 changes: 39 additions & 0 deletions projects/amdsmi/amdsmi_cli/subcommands/node.py
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,9 @@ def node(
"limit": "N/A",
"status": "N/A",
"threshold": "N/A",
"balancing_mode": "N/A",
"current_node_power": "N/A",
"supported_balancing_modes": "N/A",
}
power_unit = "W"
limit = "N/A"
Expand Down Expand Up @@ -111,6 +113,30 @@ def node(
if current_node_power != "N/A":
npm_dict["current_node_power"] = current_node_power

# Balancing mode queries are not gated on NPM being enabled, so
# they run regardless of whether npm_info came back as a dict.
if args.nodes is not None:
try:
mode = amdsmi_interface.amdsmi_get_npm_balancing_mode(args.nodes)
npm_dict["balancing_mode"] = self.helpers.NPM_BALANCING_MODE_TO_CLI.get(
mode, mode
)
except amdsmi_exception.AmdSmiLibraryException as e:
logging.debug("amdsmi_get_npm_balancing_mode failed: %s", e.get_error_info())

try:
supported_modes = amdsmi_interface.amdsmi_get_npm_supported_balancing_modes(
args.nodes
)
npm_dict["supported_balancing_modes"] = [
self.helpers.NPM_BALANCING_MODE_TO_CLI.get(mode, mode)
for mode in supported_modes
]
except amdsmi_exception.AmdSmiLibraryException as e:
logging.debug(
"amdsmi_get_npm_supported_balancing_modes failed: %s", e.get_error_info()
)

# Get base board temperatures using node_handle
if args.base_board_temps:
if args.nodes is not None:
Expand Down Expand Up @@ -165,8 +191,16 @@ def node(
node_output.append(f" STATUS: {npm_dict.get('status', 'N/A')}")
threshold = npm_dict.get("threshold", "N/A")
node_output.append(f" THRESHOLD: {threshold} {power_unit}")
balancing_mode = npm_dict.get("balancing_mode", "N/A")
node_output.append(f" BALANCING_MODE: {balancing_mode}")
current_node_power = npm_dict.get("current_node_power", "N/A")
node_output.append(f" CURRENT_NODE_POWER: {current_node_power} {power_unit}")
supported_balancing_modes = npm_dict.get("supported_balancing_modes", "N/A")
if isinstance(supported_balancing_modes, list):
supported_balancing_modes = ", ".join(supported_balancing_modes)
node_output.append(
f" SUPPORTED_BALANCING_MODES: {supported_balancing_modes}"
)
if args.base_board_temps and base_board_temp_dict:
node_output.append(" BASEBOARD:")
node_output.append(" TEMPERATURE:")
Expand Down Expand Up @@ -199,7 +233,12 @@ def node(
csv_dict["limit"] = npm_dict.get("limit", "N/A")
csv_dict["status"] = npm_dict.get("status", "N/A")
csv_dict["threshold"] = npm_dict.get("threshold", "N/A")
csv_dict["balancing_mode"] = npm_dict.get("balancing_mode", "N/A")
csv_dict["current_node_power"] = npm_dict.get("current_node_power", "N/A")
supported_balancing_modes = npm_dict.get("supported_balancing_modes", "N/A")
if isinstance(supported_balancing_modes, list):
supported_balancing_modes = ";".join(supported_balancing_modes)
csv_dict["supported_balancing_modes"] = supported_balancing_modes
if args.base_board_temps and base_board_temp_dict:
csv_dict.update(base_board_temp_dict)
if args.gtt and gtt_dict:
Expand Down
22 changes: 21 additions & 1 deletion projects/amdsmi/amdsmi_cli/subcommands/set_value.py
Original file line number Diff line number Diff line change
Expand Up @@ -2075,6 +2075,7 @@ def set_value(
core_msr_floor_limit=None,
mem_carveout=None,
gtt=None,
node_balancing_mode=None,
):
"""Issue reset commands to target gpu(s)

Expand Down Expand Up @@ -2120,7 +2121,7 @@ def set_value(
process_isolation (int, optional): Value override for args.process_isolation. Defaults to None.
Raises:
AmdSmiRequiredCommandException: If no device target or argument is provided
AmdSmiInvalidParameterException: If GPU/CPU/CORE arguments are combined or --gtt is misused
AmdSmiInvalidParameterException: If GPU/CPU/CORE arguments are combined or --gtt/--node-balancing-mode is misused
PermissionError: If a set operation requires elevation (AMDSMI_STATUS_NO_PERM)

Return:
Expand All @@ -2134,6 +2135,8 @@ def set_value(
args.cpu = cpu
if core:
args.core = core
if node_balancing_mode:
args.node_balancing_mode = node_balancing_mode

# Special GTT handling (system-wide, not per-GPU) — handle before device dispatch
if hasattr(args, "gtt") and args.gtt is not None:
Expand Down Expand Up @@ -2169,6 +2172,23 @@ def set_value(
self.logger.print_output()
return

# Special node balancing mode handling (system-wide, not per-GPU) — handle before device dispatch
if hasattr(args, "node_balancing_mode") and args.node_balancing_mode is not None:
if hasattr(args, "gpu") and args.gpu is not None:
msg = (
"amd-smi set: error: argument --node-balancing-mode: not allowed with argument "
"--gpu/-g (--node-balancing-mode is a system-wide setting, not per-GPU)"
)
raise AmdSmiInvalidParameterException(
"set", "--node-balancing-mode", self.helpers.get_output_format(), msg
)
result = self.helpers.validate_and_set_node_balancing_mode(
self.node_handle, args.node_balancing_mode, self.logger
)
self.logger.output["set_node_balancing_mode"] = result
self.logger.print_output()
return

# Special node power limit handling (node-wide, not per-GPU) — handle before device dispatch
if hasattr(args, "node_power_limit") and args.node_power_limit is not None:
if hasattr(args, "gpu") and args.gpu is not None:
Expand Down
17 changes: 17 additions & 0 deletions projects/amdsmi/docs/how-to/amdsmi-cli-tool.md
Original file line number Diff line number Diff line change
Expand Up @@ -668,6 +668,9 @@ Set Arguments:
-R, --process-isolation STATUS Enable or disable the GPU process isolation on a per partition basis: 0 for disable and 1 for enable.
--ptl-status STATUS Enable or disable the PTL on a GPU processor: 0 for disable and 1 for enable
--ptl-format FRMT1,FRMT2 Set the PTL format on a GPU processor. For example, --ptl-format I8,F32
--node-balancing-mode {POWER_BALANCING,FREQUENCY_BALANCING}
Set NPM balancing mode: POWER_BALANCING or FREQUENCY_BALANCING.
This is a system-wide setting, not per-GPU.
-n, --node-power-limit WATTS Set the node-level (NPM) power limit in watts.
This is a node-wide setting, not per-GPU.
Max node power limit: 6000 W
Expand Down Expand Up @@ -1072,6 +1075,20 @@ On systems without UALoE hardware/session, `amdsmi_get_tray_info()` returns
`AMDSMI_STATUS_NOT_SUPPORTED` and the `TRAY:` block (and the `tray`/
`max_acc_per_tray`/`tray_type` keys in `--json`/`--csv`) is omitted entirely.

`amd-smi node --power-management` includes `BALANCING_MODE`, the NPM
balancing mode (`POWER_BALANCING`, the default, or `FREQUENCY_BALANCING`).
This field reads `N/A` when the underlying value is missing or unreadable,
not when NPM is disabled (see `STATUS` for enablement). Use
`amd-smi set --node-balancing-mode {POWER_BALANCING,FREQUENCY_BALANCING}` to
change it (AMD-SMI-only; not exposed via BMC Redfish/APML).

It also includes `SUPPORTED_BALANCING_MODES`, the balancing modes supported by
this node's platform/ASIC (e.g. `POWER_BALANCING, FREQUENCY_BALANCING`),
independent of current NPM enablement. `amd-smi set --node-balancing-mode`
returns `AMDSMI_STATUS_SETTING_UNAVAILABLE` with a message naming the
requested mode if it is absent from this set, distinct from
`AMDSMI_STATUS_NOT_SUPPORTED`'s "NPM is disabled on this node" message.

## Interpreting the output

When you run an `amd-smi` command, the tool presents detailed information
Expand Down
Loading
Loading